Oct 30 12:33:19.011843 ip-10-0-137-228 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 12:33:19.011858 ip-10-0-137-228 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 12:33:19.011869 ip-10-0-137-228 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 12:33:19.012221 ip-10-0-137-228 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 12:33:29.260747 ip-10-0-137-228 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 12:33:29.260765 ip-10-0-137-228 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e1c59c6590354538a947a0192302a5c6 --
Oct 30 12:35:39.516397 ip-10-0-137-228 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 12:35:39.958414 ip-10-0-137-228 kubenswrapper[2587]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:35:39.958414 ip-10-0-137-228 kubenswrapper[2587]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 12:35:39.958414 ip-10-0-137-228 kubenswrapper[2587]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:35:39.958414 ip-10-0-137-228 kubenswrapper[2587]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:35:39.964368 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964274    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:39.964368 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964368    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:39.964368 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964372    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:39.964368 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964375    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964378    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964381    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964383    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964386    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964388    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964393    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964397    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964400    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964402    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964405    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964408    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964410    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964413    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964417    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964419    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964422    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964424    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964427    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964429    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:39.964492 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964432    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964434    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964437    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964439    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964442    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964445    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964447    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964450    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964453    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964455    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964458    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964460    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964463    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964467    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964470    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964472    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964475    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964477    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964480    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:39.964959 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964483    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964485    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964489    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964492    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964495    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964498    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964501    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964504    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964507    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964509    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964512    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964514    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964517    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964519    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964522    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964525    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964527    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964530    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964532    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964535    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:39.965454 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964537    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964539    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964542    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964544    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964547    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964550    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964553    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964556    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964559    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964562    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964564    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964567    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964571    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964575    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964578    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964581    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964584    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964586    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964590    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:39.966043 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964593    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964596    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964598    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964601    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964603    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964606    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964608    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964611    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964613    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964616    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964618    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964621    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964624    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964626    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964629    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964632    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964634    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964637    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964640    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:39.966487 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964643    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964646    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964649    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964651    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964654    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.964656    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965508    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965515    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965518    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965521    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965524    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965526    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965529    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965532    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965536    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965539    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965542    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965544    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965547    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965550    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:39.966938 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965552    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965555    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965558    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965560    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965563    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965566    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965568    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965571    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965574    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965576    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965579    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965583    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965586    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965588    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965591    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965594    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965597    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965599    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965601    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965604    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:39.967431 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965607    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965609    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965612    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965614    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965617    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965619    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965622    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965625    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965628    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965630    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965633    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965635    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965638    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965640    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965643    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965645    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965648    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965650    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965653    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965657    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:39.967943 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965659    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965664    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965668    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965671    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965675    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965678    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965680    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965683    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965685    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965688    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965691    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965694    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965697    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965699    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965702    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965704    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965706    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965709    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965712    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:39.968447 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965715    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965717    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965720    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965722    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965725    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965727    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965730    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965733    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965735    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965738    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965741    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965744    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965746    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965749    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965751    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965754    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965757    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965759    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965762    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965764    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:39.968893 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965766    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965769    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965771    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965773    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965776    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965779    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965783    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965787    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965790    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965797    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965801    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965804    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.965806    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966357    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966363    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966367    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966370    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966372    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966375    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:39.969381 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966377    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966380    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966383    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966385    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966388    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966390    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966393    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966395    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966398    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966401    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966404    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966406    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966409    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966411    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966414    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966416    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966419    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966422    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966424    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:39.969835 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966426    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966429    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966431    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966437    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966439    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966441    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966444    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966447    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966450    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966452    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966455    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966457    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966460    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966462    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966465    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966467    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966470    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966472    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966475    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966478    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:39.970296 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966480    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966483    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966487    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966489    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966492    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966495    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966498    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966501    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966503    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966506    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966508    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966511    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966513    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966516    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966518    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966522    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966524    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966527    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966529    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966532    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:39.970784 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966535    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966537    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966539    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966542    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966545    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966547    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966550    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966552    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966554    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966557    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966559    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966562    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966564    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966566    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966572    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966576    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966579    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966582    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966585    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:39.971264 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966588    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966591    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966594    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966596    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966599    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966601    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966604    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966607    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966609    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966612    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966614    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966619    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966623    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966626    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966629    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966631    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966634    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966637    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966639    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966641    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:39.971713 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966644    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.966646    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968170    2587 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968182    2587 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968190    2587 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968195    2587 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968200    2587 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968204    2587 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968209    2587 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968213    2587 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968217    2587 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968220    2587 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968223    2587 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968227    2587 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968230    2587 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968233    2587 flags.go:64] FLAG: --cgroup-root=""
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968236    2587 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968239    2587 flags.go:64] FLAG: --client-ca-file=""
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968242    2587 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968245    2587 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968250    2587 flags.go:64] FLAG: --cluster-domain=""
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968253    2587 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968257    2587 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968260    2587 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 12:35:39.972208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968263    2587 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968267    2587 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968270    2587 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968274    2587 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968277    2587 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968280    2587 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968283    2587 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968286    2587 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968289    2587 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968292    2587 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968298    2587 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968301    2587 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968304    2587 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968307    2587 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968310    2587 flags.go:64] FLAG: --enable-server="true"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968313    2587 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968317    2587 flags.go:64] FLAG: --event-burst="100"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968320    2587 flags.go:64] FLAG: --event-qps="50"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968323    2587 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968327    2587 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968330    2587 flags.go:64] FLAG: --eviction-hard=""
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968334    2587 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968338    2587 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968341    2587 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968344    2587 flags.go:64] FLAG: --eviction-soft=""
Oct 30 12:35:39.972765 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968347    2587 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968350    2587 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968353    2587 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968355    2587 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968358    2587 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968361    2587 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968364    2587 flags.go:64] FLAG: --feature-gates=""
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968368    2587 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968371    2587 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968375    2587 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968378    2587 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968382    2587 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968385    2587 flags.go:64] FLAG: --help="false"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968388    2587 flags.go:64] FLAG: --hostname-override="ip-10-0-137-228.ec2.internal"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968391    2587 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968394    2587 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968397    2587 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968400    2587 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968404    2587 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968407    2587 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968410    2587 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968414    2587 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968416    2587 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 12:35:39.973442 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968419    2587 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968422    2587 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968425    2587 flags.go:64] FLAG: --kube-reserved=""
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968428    2587 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968431    2587 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968437    2587 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968440    2587 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968443    2587 flags.go:64] FLAG: --lock-file=""
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968445    2587 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968449    2587 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968452    2587 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968458    2587 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968461    2587 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968464    2587 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968467    2587 flags.go:64] FLAG: --logging-format="text"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968470    2587 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968473    2587 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968476    2587 flags.go:64] FLAG: --manifest-url=""
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968479    2587 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968490    2587 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968494    2587 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968502    2587 flags.go:64] FLAG: --max-pods="110"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968506    2587 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968509    2587 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968512    2587 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 12:35:39.974017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968515    2587 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968518    2587 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968521    2587 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968523    2587 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968531    2587 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968534    2587 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968537    2587 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968540    2587 flags.go:64] FLAG: --pod-cidr=""
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968543    2587 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968546    2587 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968549    2587 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968552    2587 flags.go:64] FLAG: --port="10250"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968556    2587 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968560    2587 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0df263eda56ea87ab"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968563    2587 flags.go:64] FLAG: --qos-reserved=""
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968566    2587 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968569    2587 flags.go:64] FLAG: --register-node="true"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968572    2587 flags.go:64] FLAG: --register-with-taints=""
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968576    2587 flags.go:64] FLAG: --registry-burst="10"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968579    2587 flags.go:64] FLAG: --registry-qps="5"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968582    2587 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968585    2587 flags.go:64] FLAG: --reserved-memory=""
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968589    2587 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968592    2587 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968595    2587 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968598    2587 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 12:35:39.974603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968601    2587 flags.go:64] FLAG: --runonce="false"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968604    2587 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968608    2587 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968611    2587 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968614    2587 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968617    2587 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968620    2587 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968623    2587 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968626    2587 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968629    2587 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968634    2587 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968637    2587 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968640    2587 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968643    2587 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968646    2587 flags.go:64] FLAG: --system-cgroups=""
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968649    2587 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968655    2587 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968658    2587 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968662    2587 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968666    2587 flags.go:64] FLAG: --tls-min-version=""
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968670    2587 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968673    2587 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968676    2587 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968679    2587 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968682    2587 flags.go:64] FLAG: --v="2"
Oct 30 12:35:39.975257 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968686    2587 flags.go:64] FLAG: --version="false"
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968690    2587 flags.go:64] FLAG: --vmodule=""
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968695    2587 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.968698    2587 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968838    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968843    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968846    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968849    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968851    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968854    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968857    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968860    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968863    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968865    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968868    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968871    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968873    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968877    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968879    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968882    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:39.975899 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968884    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968887    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968889    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968892    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968894    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968899    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968901    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968904    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968907    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968910    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968913    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968916    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968919    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968921    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968924    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968927    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968929    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968931    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968934    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:39.976418 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968937    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968939    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968942    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968944    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968947    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968950    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968952    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968955    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968957    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968960    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968964    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968967    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968969    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968972    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968975    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968977    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968993    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.968996    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969000    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969003    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:39.976905 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969005    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969008    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969011    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969013    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969016    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969019    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969021    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969024    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969026    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969029    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969032    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969034    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969037    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969039    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969042    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969044    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969046    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969049    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969052    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:39.977596 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969054    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969057    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969059    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969063    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969066    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969069    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969071    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969073    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969076    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969078    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969081    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969084    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969087    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969091    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969095    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969098    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969102    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969105    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969108    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:39.978258 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969111    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:39.978709 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969114    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:39.978709 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969117    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:39.978709 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969120    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:39.978709 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969122    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:39.978709 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969125    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:39.978709 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969127    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:39.978709 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969130    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:39.978709 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969133    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:39.978709 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969137    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:39.978709 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969140    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:39.978709 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969143    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:39.978709 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969146    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:39.978709 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.969175    2587 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:35:39.978709 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969212    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:39.978709 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969217    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969221    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969224    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969227    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969229    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969232    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969234    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969237    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969239    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969242    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969246    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969248    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969251    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969254    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969256    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969259    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969261    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969265    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969267    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969270    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:39.979091 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969273    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969275    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969278    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969281    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969283    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969286    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969288    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969291    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969293    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969296    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969298    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969301    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969304    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969308    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969311    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969313    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969315    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969318    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969320    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:39.979553 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969323    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969325    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969328    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969331    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969334    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969336    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969339    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969341    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969344    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969346    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969349    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969352    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969354    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969358    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969362    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969365    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969368    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969371    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969373    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:39.980017 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969375    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969378    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969382    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969385    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969387    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969390    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969393    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969397    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969400    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969404    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969407    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969410    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969412    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969415    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969418    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969421    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969424    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969426    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969429    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:39.980481 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969431    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969433    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969436    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969438    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969441    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969443    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969446    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969448    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969451    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969453    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969455    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969458    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969460    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969462    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969465    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969467    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969470    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969472    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969474    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969478    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:39.980933 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969482    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969485    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969487    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969490    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969492    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969495    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969497    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.969500    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.978555    2587 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.978574    2587 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.979365    2587 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.979371    2587 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979434    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979440    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979444    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979448    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979453    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979455    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979458    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979462    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:39.981425 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979465    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979468    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979471    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979474    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979476    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979479    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979482    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979484    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979488    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979490    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979493    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979495    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979498    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979500    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979503    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979506    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979510    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979514    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979518    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:39.981881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979522    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979525    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979529    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979533    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979537    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979541    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979544    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979547    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979549    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979551    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979554    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979557    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979560    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979563    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979565    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979568    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979571    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979573    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979576    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:39.982419 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979579    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979581    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979584    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979587    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979589    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979592    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979594    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979597    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979599    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979601    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979604    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979606    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979609    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979611    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979615    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979618    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979621    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979623    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979626    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979628    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:39.982881 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979631    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979633    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979636    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979638    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979641    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979643    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979646    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979648    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979651    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979653    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979656    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979658    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979661    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979664    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979667    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979669    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979672    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979675    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979677    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:39.983389 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979680    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979682    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979684    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979687    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979689    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979692    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979694    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979697    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979700    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979702    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979705    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979708    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979710    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979713    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979715    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979718    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979720    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979723    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979725    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979728    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:39.983833 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979730    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.979760    2587 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979910    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979916    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979919    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979922    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979924    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979928    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979930    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979933    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979935    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979937    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979940    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979943    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979945    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979948    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:39.984337 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979950    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979952    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979955    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979959    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979964    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979967    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979970    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979972    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979975    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979978    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979993    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979996    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.979999    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980001    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980004    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980007    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980009    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980012    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980014    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:39.984720 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980017    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980019    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980032    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980035    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980039    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980044    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980047    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980051    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980053    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980056    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980059    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980062    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980064    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980067    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980069    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980072    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980074    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980077    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980080    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:39.985185 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980082    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980085    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980088    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980090    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980093    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980095    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980098    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980101    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980103    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980106    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980109    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980111    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980114    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980116    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980119    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980121    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980123    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980126    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980128    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:39.985637 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980132    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980134    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980137    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980140    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980142    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980145    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980148    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980150    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980153    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980155    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980158    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980161    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980164    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980166    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980168    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980171    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980173    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980176    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980178    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:39.986240 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980181    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980183    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980186    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980188    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980191    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980193    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980195    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980198    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980200    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980203    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980205    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980208    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980210    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980212    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980215    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980218    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:39.986691 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.980243    2587 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980277    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980283    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980286    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980288    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980291    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980294    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980297    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980299    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980302    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980305    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980308    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980310    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980313    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980315    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980318    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980320    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980323    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980325    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:39.987078 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980328    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980330    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980333    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980336    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980338    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980341    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980343    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980346    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980349    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980351    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980354    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980356    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980359    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980362    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980365    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980371    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980373    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980376    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980379    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:39.987519 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980382    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980385    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980388    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980390    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980393    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980395    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980398    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980400    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980403    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980405    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980408    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980410    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980413    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980415    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980418    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980420    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980423    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980425    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980428    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980430    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:39.987969 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980433    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980435    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980437    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980440    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980442    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980445    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980448    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980450    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980455    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980457    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980460    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980462    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980466    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980471    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980475    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980478    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980481    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980484    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980486    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:39.988468 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980489    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980492    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980495    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980498    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980501    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980503    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980506    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980509    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980511    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980514    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980516    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980519    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980521    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980524    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980528    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980531    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980533    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980537    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980540    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:39.988941 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980543    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980546    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980550    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980553    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980555    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980558    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980562    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980565    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980567    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980570    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:39.980572    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.980628    2587 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.985368    2587 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.986667    2587 server.go:1014] "Starting client certificate rotation"
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.986768    2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.986815    2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:35:39.989445 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.988800    2587 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:35:39.992002 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:39.991974    2587 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:35:40.000621 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.000597    2587 log.go:25] "Validated CRI v1 runtime API"
Oct 30 12:35:40.004312 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.004293    2587 log.go:25] "Validated CRI v1 image API"
Oct 30 12:35:40.005191 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.005168    2587 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 12:35:40.009787 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.009765    2587 fs.go:135] Filesystem UUIDs: map[4f9c305b-8994-4260-987b-17215f91db9a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d8bc8d32-21b6-4c15-92a6-db6946fbc4cb:/dev/nvme0n1p3]
Oct 30 12:35:40.009869 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.009786    2587 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 12:35:40.015435 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.015314    2587 manager.go:217] Machine: {Timestamp:2026-10-30 12:35:40.013937173 +0000 UTC m=+0.364236633 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3155175 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25b0a17f88fcfef32e0bd984b33381 SystemUUID:ec25b0a1-7f88-fcfe-f32e-0bd984b33381 BootID:e1c59c65-9035-4538-a947-a0192302a5c6 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:ac:18:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:ac:18:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:4b:4b:4c:e1:80 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 12:35:40.015435 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.015433    2587 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 12:35:40.015584 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.015572    2587 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 12:35:40.018725 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.018539    2587 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 12:35:40.018872 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.018729    2587 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-228.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 12:35:40.019509 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.019497    2587 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 12:35:40.019540 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.019511    2587 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 12:35:40.019540 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.019525    2587 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:35:40.020501 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.020490    2587 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:35:40.021390 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.021380    2587 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 12:35:40.024560 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.024547    2587 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 12:35:40.024710 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.024701    2587 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 12:35:40.025385 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.025373    2587 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 12:35:40.025415 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.025392    2587 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 12:35:40.025415 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.025403    2587 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 12:35:40.025415 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.025412    2587 kubelet.go:405] "Adding apiserver pod source"
Oct 30 12:35:40.025511 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.025421    2587 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 12:35:40.025815 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.025797    2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:35:40.028472 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.028451    2587 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 12:35:40.034323 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.034308    2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:35:40.035128 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.035116    2587 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 12:35:40.035166 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.035147    2587 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 12:35:40.035313 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.035303    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 12:35:40.035361 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.035319    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 12:35:40.035361 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.035325    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 12:35:40.035361 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.035331    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 12:35:40.035361 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.035336    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 12:35:40.035361 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.035345    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 12:35:40.035927 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.035919    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 12:35:40.035956 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.035929    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 12:35:40.035956 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.035936    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 12:35:40.035956 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.035942    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 12:35:40.035956 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.035950    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 12:35:40.036105 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.035959    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 12:35:40.036894 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.036885    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 12:35:40.036925 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.036895    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 12:35:40.040943 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.040927    2587 server.go:1263] "Started kubelet"
Oct 30 12:35:40.041074 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.041046    2587 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 12:35:40.041136 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.041039    2587 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 12:35:40.041136 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.041121    2587 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 12:35:40.041808 ip-10-0-137-228 systemd[1]: Started Kubernetes Kubelet.
Oct 30 12:35:40.042339 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.042310    2587 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 12:35:40.043211 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.043198    2587 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 12:35:40.047885 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:40.047851    2587 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 12:35:40.048378 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.048360    2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 12:35:40.049128 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.049107    2587 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 12:35:40.049227 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:40.049198    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-228.ec2.internal\" not found"
Oct 30 12:35:40.049681 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.049663    2587 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 12:35:40.049771 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.049685    2587 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 12:35:40.049771 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.049665    2587 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 12:35:40.049854 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.049785    2587 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 12:35:40.049854 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.049793    2587 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 12:35:40.050049 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.049856    2587 factory.go:55] Registering systemd factory
Oct 30 12:35:40.050049 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.049872    2587 factory.go:223] Registration of the systemd container factory successfully
Oct 30 12:35:40.050154 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.050100    2587 factory.go:153] Registering CRI-O factory
Oct 30 12:35:40.050154 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.050112    2587 factory.go:223] Registration of the crio container factory successfully
Oct 30 12:35:40.050154 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.050153    2587 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 12:35:40.050274 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.050169    2587 factory.go:103] Registering Raw factory
Oct 30 12:35:40.050274 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.050181    2587 manager.go:1199] Started watching for new ooms in manager
Oct 30 12:35:40.051216 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.051203    2587 manager.go:319] Starting recovery of all containers
Oct 30 12:35:40.062678 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.062659    2587 manager.go:324] Recovery completed
Oct 30 12:35:40.066941 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.066925    2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:35:40.068943 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.068907    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-228.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:35:40.069047 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.068957    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-228.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:35:40.069047 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.068970    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-228.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:35:40.069565 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.069550    2587 cpu_manager.go:226] "Starting" policy="none"
Oct 30 12:35:40.069565 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.069560    2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:35:40.069681 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.069564    2587 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 12:35:40.069681 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.069599    2587 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 12:35:40.070269 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.070253    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-228.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:35:40.070340 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.070280    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-228.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:35:40.070340 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.070290    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-228.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:35:40.071112 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.071100    2587 policy_none.go:50] "Start"
Oct 30 12:35:40.071149 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.071116    2587 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 12:35:40.071149 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.071127    2587 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 12:35:40.072337 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.072325    2587 policy_none.go:44] "Start"
Oct 30 12:35:40.073420 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.073391    2587 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 12:35:40.073511 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.073474    2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qn2x6"
Oct 30 12:35:40.074054 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:40.074032    2587 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 12:35:40.074145 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.074034    2587 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-228.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 12:35:40.075060 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:40.074026    2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-228.ec2.internal.18e34ed1e63417ae  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-228.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-228.ec2.internal,},FirstTimestamp:2026-10-30 12:35:40.040894382 +0000 UTC m=+0.391193842,LastTimestamp:2026-10-30 12:35:40.040894382 +0000 UTC m=+0.391193842,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-228.ec2.internal,}"
Oct 30 12:35:40.077888 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.077873    2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qn2x6"
Oct 30 12:35:40.112807 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.112764    2587 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 12:35:40.115635 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.115608    2587 manager.go:341] "Starting Device Plugin manager"
Oct 30 12:35:40.115721 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:40.115652    2587 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 12:35:40.115721 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.115662    2587 server.go:87] "Starting device plugin registration server"
Oct 30 12:35:40.116070 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.115901    2587 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 12:35:40.116106 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.116079    2587 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 12:35:40.116204 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.116186    2587 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 12:35:40.116329 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.116317    2587 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 12:35:40.116329 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.116329    2587 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 12:35:40.116826 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:40.116801    2587 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 12:35:40.116907 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:40.116864    2587 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-228.ec2.internal\" not found"
Oct 30 12:35:40.121247 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.121222    2587 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:40.123567 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.123551    2587 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:40.164859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.153897    2587 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:40.198426 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.198394    2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 12:35:40.199788 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.199765    2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 12:35:40.199788 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.199789    2587 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 12:35:40.199943 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.199818    2587 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 12:35:40.199943 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:40.199852    2587 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 12:35:40.215552 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.215480    2587 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:40.216466 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.216451    2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:35:40.217475 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.217460    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-228.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:35:40.217584 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.217517    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-228.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:35:40.217584 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.217534    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-228.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:35:40.217584 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.217564    2587 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.238998 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.238962    2587 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.239136 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:40.239015    2587 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-228.ec2.internal\": node \"ip-10-0-137-228.ec2.internal\" not found"
Oct 30 12:35:40.301031 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.300951    2587 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal"]
Oct 30 12:35:40.302781 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.302764    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.303959 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.303943    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.327351 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.327329    2587 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.330771 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.330753    2587 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.351938 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.351912    2587 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:35:40.353060 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.353036    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6068dd25b70d52ed49d0937c3f3549a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal\" (UID: \"6068dd25b70d52ed49d0937c3f3549a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.353171 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.353075    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6068dd25b70d52ed49d0937c3f3549a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal\" (UID: \"6068dd25b70d52ed49d0937c3f3549a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.353171 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.353100    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72d3c22f1a6da7d8f3667e690d2cfffa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-228.ec2.internal\" (UID: \"72d3c22f1a6da7d8f3667e690d2cfffa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.355233 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.355219    2587 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:35:40.453906 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.453868    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6068dd25b70d52ed49d0937c3f3549a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal\" (UID: \"6068dd25b70d52ed49d0937c3f3549a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.453906 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.453904    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6068dd25b70d52ed49d0937c3f3549a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal\" (UID: \"6068dd25b70d52ed49d0937c3f3549a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.454147 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.453928    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72d3c22f1a6da7d8f3667e690d2cfffa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-228.ec2.internal\" (UID: \"72d3c22f1a6da7d8f3667e690d2cfffa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.454147 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.454008    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6068dd25b70d52ed49d0937c3f3549a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal\" (UID: \"6068dd25b70d52ed49d0937c3f3549a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.454147 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.454008    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6068dd25b70d52ed49d0937c3f3549a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal\" (UID: \"6068dd25b70d52ed49d0937c3f3549a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.454147 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.454008    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72d3c22f1a6da7d8f3667e690d2cfffa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-228.ec2.internal\" (UID: \"72d3c22f1a6da7d8f3667e690d2cfffa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.656929 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.656846    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.659330 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.659306    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:35:40.986208 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.986108    2587 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 12:35:40.986785 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.986274    2587 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 12:35:40.986785 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.986305    2587 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 12:35:40.986785 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:40.986312    2587 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 12:35:41.026337 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.026301    2587 apiserver.go:52] "Watching apiserver"
Oct 30 12:35:41.048483 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.048457    2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 12:35:41.063718 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.063695    2587 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 12:35:41.064116 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.064079    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-9trzv","kube-system/konnectivity-agent-n6d9z","kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal","openshift-cluster-node-tuning-operator/tuned-jsgw4","openshift-dns/node-resolver-84tjf","openshift-multus/network-metrics-daemon-v4ttl","openshift-network-diagnostics/network-check-target-chqf2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v","openshift-image-registry/node-ca-p9gs7","openshift-network-operator/iptables-alerter-vmqth","kube-system/global-pull-secret-syncer-kd88h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal","openshift-multus/multus-5dfvg","openshift-multus/multus-additional-cni-plugins-x6hvr"]
Oct 30 12:35:41.066222 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.066203    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.067833 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.067808    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:35:41.069617 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.069597    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:41.071578 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.071550    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:35:41.072971 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.072948    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:41.073118 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:41.073094    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:35:41.074417 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.074391    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:41.074493 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:41.074474    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:35:41.075807 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.075791    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:41.077287 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.077265    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:35:41.078690 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.078670    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:35:41.081029 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.080976    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:41.081206 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.081184    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 12:35:41.081368 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.081353    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-scxms\""
Oct 30 12:35:41.082047 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.081636    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 12:35:41.082313 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.082233    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 12:35:41.082389 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.082351    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6qrlt\""
Oct 30 12:35:41.083353 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.082634    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 12:35:41.083353 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.082723    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 12:35:41.083353 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.082934    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x6m2k\""
Oct 30 12:35:41.083353 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.083079    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 12:35:41.083353 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.083094    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:35:41.083353 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:41.081178    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:35:41.083353 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.083219    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 12:35:41.083353 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.083239    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 12:35:41.083353 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.083264    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:30:41 +0000 UTC" deadline="2028-04-06 23:34:20.414110233 +0000 UTC"
Oct 30 12:35:41.083353 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.083321    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12586h58m39.330792494s"
Oct 30 12:35:41.084777 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.084017    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 12:35:41.085712 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.085690    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.086700 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.086632    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 12:35:41.088418 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.087225    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 12:35:41.088418 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.087672    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:41.089721 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.089710    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2vwzh\""
Oct 30 12:35:41.092148 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.092128    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 12:35:41.092349 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.092331    2587 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 12:35:41.150791 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.150724    2587 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 12:35:41.156588 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.156563    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-var-lib-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.156717 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.156592    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bbb54d63-c3cc-4759-ba36-252e1ffc4d18-agent-certs\") pod \"konnectivity-agent-n6d9z\" (UID: \"bbb54d63-c3cc-4759-ba36-252e1ffc4d18\") " pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:35:41.156717 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.156609    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovnkube-config\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.156717 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.156624    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h77bx\" (UniqueName: \"kubernetes.io/projected/c72d4793-688e-4c37-bb51-e0a9100a2203-kube-api-access-h77bx\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:35:41.156717 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.156642    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/233c3bd3-01bc-4c71-8bc4-611933e34539-iptables-alerter-script\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:35:41.156717 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.156683    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-modprobe-d\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:41.156717 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.156714    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:41.156899 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.156734    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-cni-multus\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.156899 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.156782    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-kubelet\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.156899 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.156821    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-ovn\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.156899 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.156847    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovnkube-script-lib\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.156899 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.156884    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-cnibin\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.157093 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.156926    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-daemon-config\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.157093 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.156955    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-multus-certs\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.157093 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.156971    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:41.157093 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157026    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-255mf\" (UniqueName: \"kubernetes.io/projected/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-kube-api-access-255mf\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:41.157093 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157064    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-device-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:41.157093 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157081    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c24sf\" (UniqueName: \"kubernetes.io/projected/eb31cffb-32c0-4d28-9947-325621908909-kube-api-access-c24sf\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:41.157309 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157103    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-system-cni-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.157309 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157125    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-netns\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.157309 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157147    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-system-cni-dir\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:41.157309 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157173    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-cnibin\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:41.157309 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157190    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.157309 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157205    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-cni-bin\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.157309 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157221    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:41.157309 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157250    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtx6n\" (UniqueName: \"kubernetes.io/projected/996125c2-4148-45f2-845d-33e743fdd73a-kube-api-access-xtx6n\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:41.157309 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157279    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-systemd-units\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.157309 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157304    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8gn6\" (UniqueName: \"kubernetes.io/projected/0a195064-4c82-47ec-939e-8f0ac45fb0f6-kube-api-access-h8gn6\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.157598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157329    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-sys-fs\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:41.157598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157354    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-conf-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.157598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157367    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-run-netns\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.157598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157389    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-etc-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.157598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157402    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-node-log\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.157598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157416    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bbb54d63-c3cc-4759-ba36-252e1ffc4d18-konnectivity-ca\") pod \"konnectivity-agent-n6d9z\" (UID: \"bbb54d63-c3cc-4759-ba36-252e1ffc4d18\") " pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:35:41.157598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157438    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c72d4793-688e-4c37-bb51-e0a9100a2203-serviceca\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:35:41.157598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157451    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4fe0229-c475-400a-b6c0-48d80476b3b5-tmp-dir\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:35:41.157598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157482    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-k8s-cni-cncf-io\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.157598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157497    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-socket-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:41.157598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157510    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-kubernetes\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:41.157598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157522    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb31cffb-32c0-4d28-9947-325621908909-tmp\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:41.157598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157537    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhm4h\" (UniqueName: \"kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h\") pod \"network-check-target-chqf2\" (UID: \"cd722809-68b9-41b8-8a5e-814903a74299\") " pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:41.157598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157551    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-774wq\" (UniqueName: \"kubernetes.io/projected/14b22d89-19e4-4dbb-a031-91b5fe337785-kube-api-access-774wq\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.157598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157590    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74n4f\" (UniqueName: \"kubernetes.io/projected/ca860770-e681-4dae-806f-ef6de17bed8e-kube-api-access-74n4f\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157612    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysctl-d\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157629    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-host\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157676    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-dbus\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157711    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-kubelet\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157748    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb31cffb-32c0-4d28-9947-325621908909-etc-tuned\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157795    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-cni-bin\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157825    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-sys\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157852    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-var-lib-kubelet\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157877    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-kubelet-config\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157907    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-cni-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157932    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14b22d89-19e4-4dbb-a031-91b5fe337785-cni-binary-copy\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157949    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-slash\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.157972    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-systemd\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158022    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-env-overrides\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158045    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/233c3bd3-01bc-4c71-8bc4-611933e34539-host-slash\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158069    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-socket-dir-parent\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.158308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158126    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158136    2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4l8xm"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158157    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158183    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158210    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-log-socket\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158254    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158278    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158301    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysconfig\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158357    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-hostroot\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158379    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158394    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-systemd\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158409    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158445    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-registration-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158471    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e4fe0229-c475-400a-b6c0-48d80476b3b5-hosts-file\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158496    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfhl5\" (UniqueName: \"kubernetes.io/projected/e4fe0229-c475-400a-b6c0-48d80476b3b5-kube-api-access-rfhl5\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158531    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-etc-kubernetes\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158547    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c72d4793-688e-4c37-bb51-e0a9100a2203-host\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:35:41.158859 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158561    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v766r\" (UniqueName: \"kubernetes.io/projected/233c3bd3-01bc-4c71-8bc4-611933e34539-kube-api-access-v766r\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:35:41.159386 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158574    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-run\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:41.159386 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158591    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-cni-netd\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.159386 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158606    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovn-node-metrics-cert\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:41.159386 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158621    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysctl-conf\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:41.159386 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.158635    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-lib-modules\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:41.165039 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.165020    2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4l8xm"
Oct 30 12:35:41.181319 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.181294    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 12:35:41.182007 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.181966    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 12:35:41.182183 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.182166    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 12:35:41.182183 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.182179    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kkflg\""
Oct 30 12:35:41.182619 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.182604    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 12:35:41.182688 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.182633    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 12:35:41.182688 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.182654    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 12:35:41.182784 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.182709    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-562t6\""
Oct 30 12:35:41.182784 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.182718    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qjz2q\""
Oct 30 12:35:41.182784 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.182740    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 12:35:41.182784 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.182745    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:35:41.182784 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.182746    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 12:35:41.182784 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.182771    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 12:35:41.183012 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.182833    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 12:35:41.183012 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.182898    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rkrpr\""
Oct 30 12:35:41.183012 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.182935    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gmdlh\""
Oct 30 12:35:41.183183 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.183170    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 12:35:41.183232 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.183220    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 12:35:41.183268 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.183260    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 12:35:41.186199 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:41.186169    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6068dd25b70d52ed49d0937c3f3549a5.slice/crio-560843455db2355b850de37a5d81ae51e5429266263244dfb720309e6095a59c WatchSource:0}: Error finding container 560843455db2355b850de37a5d81ae51e5429266263244dfb720309e6095a59c: Status 404 returned error can't find the container with id 560843455db2355b850de37a5d81ae51e5429266263244dfb720309e6095a59c
Oct 30 12:35:41.186716 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:41.186690    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d3c22f1a6da7d8f3667e690d2cfffa.slice/crio-0e3f71bc99acfb4c14662813364519fafe4f46385b1fbf306688db96514f26ca WatchSource:0}: Error finding container 0e3f71bc99acfb4c14662813364519fafe4f46385b1fbf306688db96514f26ca: Status 404 returned error can't find the container with id 0e3f71bc99acfb4c14662813364519fafe4f46385b1fbf306688db96514f26ca
Oct 30 12:35:41.191786 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.191771    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:35:41.203346 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.203290    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal" event={"ID":"72d3c22f1a6da7d8f3667e690d2cfffa","Type":"ContainerStarted","Data":"0e3f71bc99acfb4c14662813364519fafe4f46385b1fbf306688db96514f26ca"}
Oct 30 12:35:41.204583 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:41.204559    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal" event={"ID":"6068dd25b70d52ed49d0937c3f3549a5","Type":"ContainerStarted","Data":"560843455db2355b850de37a5d81ae51e5429266263244dfb720309e6095a59c"}
Oct 30 12:35:42.419106 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419083    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v766r\" (UniqueName: \"kubernetes.io/projected/233c3bd3-01bc-4c71-8bc4-611933e34539-kube-api-access-v766r\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:35:42.419175 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419114    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-run\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.419175 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419133    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-cni-netd\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.419175 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419149    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovn-node-metrics-cert\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.419175 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419164    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysctl-conf\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.419327 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419209    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-lib-modules\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.419327 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419251    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-var-lib-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.419327 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419270    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bbb54d63-c3cc-4759-ba36-252e1ffc4d18-agent-certs\") pod \"konnectivity-agent-n6d9z\" (UID: \"bbb54d63-c3cc-4759-ba36-252e1ffc4d18\") " pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:35:42.419327 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419293    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovnkube-config\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.419327 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419296    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysctl-conf\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.419327 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419221    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-cni-netd\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.419327 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419324    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-var-lib-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419344    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-run\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419316    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h77bx\" (UniqueName: \"kubernetes.io/projected/c72d4793-688e-4c37-bb51-e0a9100a2203-kube-api-access-h77bx\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419443    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/233c3bd3-01bc-4c71-8bc4-611933e34539-iptables-alerter-script\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419478    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-modprobe-d\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419505    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419533    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-cni-multus\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419559    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-kubelet\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419582    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-ovn\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419596    2587 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419611    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovnkube-script-lib\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419622    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-modprobe-d\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419636    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-cnibin\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419662    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-daemon-config\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419636    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-cni-multus\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:42.419664    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:42.419698 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419680    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-kubelet\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419710    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-ovn\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419734    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-cnibin\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:42.419745    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret podName:0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:42.919722781 +0000 UTC m=+2.110611058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret") pod "global-pull-secret-syncer-kd88h" (UID: "0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419769    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-multus-certs\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419794    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419817    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-255mf\" (UniqueName: \"kubernetes.io/projected/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-kube-api-access-255mf\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419864    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-device-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419888    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c24sf\" (UniqueName: \"kubernetes.io/projected/eb31cffb-32c0-4d28-9947-325621908909-kube-api-access-c24sf\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419913    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-system-cni-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419934    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-netns\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:42.419943    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.419975    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-system-cni-dir\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:42.419997    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs podName:9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b nodeName:}" failed. No retries permitted until 2026-10-30 12:35:42.919983614 +0000 UTC m=+2.110871886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs") pod "network-metrics-daemon-v4ttl" (UID: "9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420028    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-cnibin\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420016    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-system-cni-dir\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:42.420493 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420047    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovnkube-config\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420052    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420107    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-multus-certs\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420108    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-cni-bin\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420119    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/233c3bd3-01bc-4c71-8bc4-611933e34539-iptables-alerter-script\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420128    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovnkube-script-lib\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420145    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-cnibin\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420143    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420176    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtx6n\" (UniqueName: \"kubernetes.io/projected/996125c2-4148-45f2-845d-33e743fdd73a-kube-api-access-xtx6n\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420196    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-system-cni-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420198    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-systemd-units\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420226    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-systemd-units\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420250    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8gn6\" (UniqueName: \"kubernetes.io/projected/0a195064-4c82-47ec-939e-8f0ac45fb0f6-kube-api-access-h8gn6\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420275    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-sys-fs\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420294    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-device-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420081    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420319    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-conf-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.421167 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420252    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-netns\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420348    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-cni-bin\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420363    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-daemon-config\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420385    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-run-netns\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420451    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-sys-fs\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420434    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-etc-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420498    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-run-netns\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420500    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-node-log\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420524    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-node-log\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420538    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-etc-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420540    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-conf-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420605    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bbb54d63-c3cc-4759-ba36-252e1ffc4d18-konnectivity-ca\") pod \"konnectivity-agent-n6d9z\" (UID: \"bbb54d63-c3cc-4759-ba36-252e1ffc4d18\") " pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420632    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420645    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-lib-modules\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420665    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c72d4793-688e-4c37-bb51-e0a9100a2203-serviceca\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420689    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4fe0229-c475-400a-b6c0-48d80476b3b5-tmp-dir\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420711    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-k8s-cni-cncf-io\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420751    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-k8s-cni-cncf-io\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.422051 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420775    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-socket-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420796    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-kubernetes\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420816    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb31cffb-32c0-4d28-9947-325621908909-tmp\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.420889    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-kubernetes\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421001    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhm4h\" (UniqueName: \"kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h\") pod \"network-check-target-chqf2\" (UID: \"cd722809-68b9-41b8-8a5e-814903a74299\") " pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421031    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-774wq\" (UniqueName: \"kubernetes.io/projected/14b22d89-19e4-4dbb-a031-91b5fe337785-kube-api-access-774wq\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421046    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4fe0229-c475-400a-b6c0-48d80476b3b5-tmp-dir\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421173    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74n4f\" (UniqueName: \"kubernetes.io/projected/ca860770-e681-4dae-806f-ef6de17bed8e-kube-api-access-74n4f\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421200    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysctl-d\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421209    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-socket-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421244    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-host\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421271    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-dbus\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421294    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-kubelet\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421338    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb31cffb-32c0-4d28-9947-325621908909-etc-tuned\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421244    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bbb54d63-c3cc-4759-ba36-252e1ffc4d18-konnectivity-ca\") pod \"konnectivity-agent-n6d9z\" (UID: \"bbb54d63-c3cc-4759-ba36-252e1ffc4d18\") " pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421365    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-cni-bin\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421417    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-sys\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421442    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-var-lib-kubelet\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.422834 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421468    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysctl-d\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421486    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c72d4793-688e-4c37-bb51-e0a9100a2203-serviceca\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421498    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-kubelet-config\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421501    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-host\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421562    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-cni-bin\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421587    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-kubelet-config\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421618    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-dbus\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421620    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-sys\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421651    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-kubelet\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421662    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-cni-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421711    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-var-lib-kubelet\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421749    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14b22d89-19e4-4dbb-a031-91b5fe337785-cni-binary-copy\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421790    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-slash\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421793    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-cni-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421846    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-slash\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421830    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-systemd\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421893    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-systemd\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421893    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-env-overrides\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.423586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421927    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/233c3bd3-01bc-4c71-8bc4-611933e34539-host-slash\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421953    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-socket-dir-parent\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.421977    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422000    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422027    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422053    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-log-socket\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422076    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422097    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422119    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysconfig\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422141    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-hostroot\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422188    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-hostroot\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422183    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/233c3bd3-01bc-4c71-8bc4-611933e34539-host-slash\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422221    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422247    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-systemd\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422251    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422335    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422349    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:42.424162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422361    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-registration-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422386    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e4fe0229-c475-400a-b6c0-48d80476b3b5-hosts-file\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422414    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysconfig\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422431    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhl5\" (UniqueName: \"kubernetes.io/projected/e4fe0229-c475-400a-b6c0-48d80476b3b5-kube-api-access-rfhl5\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422455    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-etc-kubernetes\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422495    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c72d4793-688e-4c37-bb51-e0a9100a2203-host\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422188    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-socket-dir-parent\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422568    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c72d4793-688e-4c37-bb51-e0a9100a2203-host\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422581    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422612    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-registration-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422624    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-systemd\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422665    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422709    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-etc-kubernetes\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422877    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14b22d89-19e4-4dbb-a031-91b5fe337785-cni-binary-copy\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422909    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-env-overrides\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422956    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-log-socket\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.422670    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e4fe0229-c475-400a-b6c0-48d80476b3b5-hosts-file\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.423133    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:42.424680 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.423157    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:42.425216 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.423304    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:42.425216 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.423996    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovn-node-metrics-cert\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.425216 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.424038    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bbb54d63-c3cc-4759-ba36-252e1ffc4d18-agent-certs\") pod \"konnectivity-agent-n6d9z\" (UID: \"bbb54d63-c3cc-4759-ba36-252e1ffc4d18\") " pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:35:42.425216 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.424143    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb31cffb-32c0-4d28-9947-325621908909-tmp\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.425216 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.424285    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb31cffb-32c0-4d28-9947-325621908909-etc-tuned\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.439461 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:42.439047    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:35:42.439461 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:42.439072    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:35:42.439461 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:42.439085    2587 projected.go:196] Error preparing data for projected volume kube-api-access-fhm4h for pod openshift-network-diagnostics/network-check-target-chqf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:42.439461 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:42.439146    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h podName:cd722809-68b9-41b8-8a5e-814903a74299 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:42.939130104 +0000 UTC m=+2.130018391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fhm4h" (UniqueName: "kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h") pod "network-check-target-chqf2" (UID: "cd722809-68b9-41b8-8a5e-814903a74299") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:42.440903 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.440836    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v766r\" (UniqueName: \"kubernetes.io/projected/233c3bd3-01bc-4c71-8bc4-611933e34539-kube-api-access-v766r\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:35:42.441020 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.440968    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h77bx\" (UniqueName: \"kubernetes.io/projected/c72d4793-688e-4c37-bb51-e0a9100a2203-kube-api-access-h77bx\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:35:42.441696 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.441677    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-255mf\" (UniqueName: \"kubernetes.io/projected/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-kube-api-access-255mf\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:42.441870 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.441847    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfhl5\" (UniqueName: \"kubernetes.io/projected/e4fe0229-c475-400a-b6c0-48d80476b3b5-kube-api-access-rfhl5\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:35:42.442190 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.442169    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-774wq\" (UniqueName: \"kubernetes.io/projected/14b22d89-19e4-4dbb-a031-91b5fe337785-kube-api-access-774wq\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.442617 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.442598    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c24sf\" (UniqueName: \"kubernetes.io/projected/eb31cffb-32c0-4d28-9947-325621908909-kube-api-access-c24sf\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.442754 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.442731    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtx6n\" (UniqueName: \"kubernetes.io/projected/996125c2-4148-45f2-845d-33e743fdd73a-kube-api-access-xtx6n\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:42.443013 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.442996    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8gn6\" (UniqueName: \"kubernetes.io/projected/0a195064-4c82-47ec-939e-8f0ac45fb0f6-kube-api-access-h8gn6\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.443138 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.443121    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74n4f\" (UniqueName: \"kubernetes.io/projected/ca860770-e681-4dae-806f-ef6de17bed8e-kube-api-access-74n4f\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:42.572786 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.572751    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:35:42.578108 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.578082    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:35:42.581874 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:42.581845    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a195064_4c82_47ec_939e_8f0ac45fb0f6.slice/crio-c1699a8cb635ca88e61742e21af223617c6801b8f9d66639db3e0a89af6b497b WatchSource:0}: Error finding container c1699a8cb635ca88e61742e21af223617c6801b8f9d66639db3e0a89af6b497b: Status 404 returned error can't find the container with id c1699a8cb635ca88e61742e21af223617c6801b8f9d66639db3e0a89af6b497b
Oct 30 12:35:42.586923 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:42.586899    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb54d63_c3cc_4759_ba36_252e1ffc4d18.slice/crio-88f7490ae2fbb52873e01de2aa61d45b5d11f1f22e6ec8849f9a9d432763195e WatchSource:0}: Error finding container 88f7490ae2fbb52873e01de2aa61d45b5d11f1f22e6ec8849f9a9d432763195e: Status 404 returned error can't find the container with id 88f7490ae2fbb52873e01de2aa61d45b5d11f1f22e6ec8849f9a9d432763195e
Oct 30 12:35:42.591303 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.591283    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:35:42.595912 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.595891    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:35:42.599156 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:42.599134    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb31cffb_32c0_4d28_9947_325621908909.slice/crio-1e62a9e2b279520394ec02598bf1b5d4693b1355717d23bb9096425e1e53fbc8 WatchSource:0}: Error finding container 1e62a9e2b279520394ec02598bf1b5d4693b1355717d23bb9096425e1e53fbc8: Status 404 returned error can't find the container with id 1e62a9e2b279520394ec02598bf1b5d4693b1355717d23bb9096425e1e53fbc8
Oct 30 12:35:42.604198 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:42.604174    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fe0229_c475_400a_b6c0_48d80476b3b5.slice/crio-42d7c5b6e1a1b85ce656693813ea8f7eb392ce931e9a4e06a49a0d75bbefbfd8 WatchSource:0}: Error finding container 42d7c5b6e1a1b85ce656693813ea8f7eb392ce931e9a4e06a49a0d75bbefbfd8: Status 404 returned error can't find the container with id 42d7c5b6e1a1b85ce656693813ea8f7eb392ce931e9a4e06a49a0d75bbefbfd8
Oct 30 12:35:42.627079 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.627052    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:35:42.628121 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.628095    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:35:42.637488 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:42.637463    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca860770_e681_4dae_806f_ef6de17bed8e.slice/crio-b753ee78655981eb196701892fba5cdbb928afa5af679fdaf272a8bb977b3300 WatchSource:0}: Error finding container b753ee78655981eb196701892fba5cdbb928afa5af679fdaf272a8bb977b3300: Status 404 returned error can't find the container with id b753ee78655981eb196701892fba5cdbb928afa5af679fdaf272a8bb977b3300
Oct 30 12:35:42.638386 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:42.638359    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72d4793_688e_4c37_bb51_e0a9100a2203.slice/crio-48a7174dbcfa7e66136bd9bb60e499286f2f73f7cc0926589caade73ef8a917b WatchSource:0}: Error finding container 48a7174dbcfa7e66136bd9bb60e499286f2f73f7cc0926589caade73ef8a917b: Status 404 returned error can't find the container with id 48a7174dbcfa7e66136bd9bb60e499286f2f73f7cc0926589caade73ef8a917b
Oct 30 12:35:42.654634 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.654607    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:35:42.662368 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.662346    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5dfvg"
Oct 30 12:35:42.663537 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:42.663516    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233c3bd3_01bc_4c71_8bc4_611933e34539.slice/crio-8dbf2415cb6007a1bae0b4031c5d6363142904eb664535c9a1483fb13a3a3227 WatchSource:0}: Error finding container 8dbf2415cb6007a1bae0b4031c5d6363142904eb664535c9a1483fb13a3a3227: Status 404 returned error can't find the container with id 8dbf2415cb6007a1bae0b4031c5d6363142904eb664535c9a1483fb13a3a3227
Oct 30 12:35:42.670649 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:42.670597    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b22d89_19e4_4dbb_a031_91b5fe337785.slice/crio-b3397153b6c8da955283cddb4d3ba179f8ee3d9d90e55adb21e2dc34f94c554c WatchSource:0}: Error finding container b3397153b6c8da955283cddb4d3ba179f8ee3d9d90e55adb21e2dc34f94c554c: Status 404 returned error can't find the container with id b3397153b6c8da955283cddb4d3ba179f8ee3d9d90e55adb21e2dc34f94c554c
Oct 30 12:35:42.673791 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.673768    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:35:42.681953 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:35:42.681932    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996125c2_4148_45f2_845d_33e743fdd73a.slice/crio-a98e62006387c24317a806ce2becb38889c5ee81d1d2c62af2f8a74ff8d0bc14 WatchSource:0}: Error finding container a98e62006387c24317a806ce2becb38889c5ee81d1d2c62af2f8a74ff8d0bc14: Status 404 returned error can't find the container with id a98e62006387c24317a806ce2becb38889c5ee81d1d2c62af2f8a74ff8d0bc14
Oct 30 12:35:42.925980 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.925877    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:42.925980 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:42.925948    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:42.926197 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:42.926146    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:42.926250 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:42.926210    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs podName:9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b nodeName:}" failed. No retries permitted until 2026-10-30 12:35:43.926192235 +0000 UTC m=+3.117080506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs") pod "network-metrics-daemon-v4ttl" (UID: "9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:42.926619 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:42.926602    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:42.926717 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:42.926648    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret podName:0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:43.926635061 +0000 UTC m=+3.117523332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret") pod "global-pull-secret-syncer-kd88h" (UID: "0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:43.027127 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.027086    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhm4h\" (UniqueName: \"kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h\") pod \"network-check-target-chqf2\" (UID: \"cd722809-68b9-41b8-8a5e-814903a74299\") " pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:43.027350 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:43.027331    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:35:43.027452 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:43.027362    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:35:43.027452 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:43.027375    2587 projected.go:196] Error preparing data for projected volume kube-api-access-fhm4h for pod openshift-network-diagnostics/network-check-target-chqf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:43.027583 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:43.027468    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h podName:cd722809-68b9-41b8-8a5e-814903a74299 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:44.027449621 +0000 UTC m=+3.218337901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fhm4h" (UniqueName: "kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h") pod "network-check-target-chqf2" (UID: "cd722809-68b9-41b8-8a5e-814903a74299") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:43.225870 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.225725    2587 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:43.326909 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.326836    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:30:42 +0000 UTC" deadline="2028-08-05 02:42:36.653327978 +0000 UTC"
Oct 30 12:35:43.326909 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.326906    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15470h6m53.326426609s"
Oct 30 12:35:43.377802 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.377761    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5dfvg" event={"ID":"14b22d89-19e4-4dbb-a031-91b5fe337785","Type":"ContainerStarted","Data":"b3397153b6c8da955283cddb4d3ba179f8ee3d9d90e55adb21e2dc34f94c554c"}
Oct 30 12:35:43.404378 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.404326    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vmqth" event={"ID":"233c3bd3-01bc-4c71-8bc4-611933e34539","Type":"ContainerStarted","Data":"8dbf2415cb6007a1bae0b4031c5d6363142904eb664535c9a1483fb13a3a3227"}
Oct 30 12:35:43.422066 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.421985    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p9gs7" event={"ID":"c72d4793-688e-4c37-bb51-e0a9100a2203","Type":"ContainerStarted","Data":"48a7174dbcfa7e66136bd9bb60e499286f2f73f7cc0926589caade73ef8a917b"}
Oct 30 12:35:43.437219 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.437122    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v" event={"ID":"ca860770-e681-4dae-806f-ef6de17bed8e","Type":"ContainerStarted","Data":"b753ee78655981eb196701892fba5cdbb928afa5af679fdaf272a8bb977b3300"}
Oct 30 12:35:43.445826 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.445776    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n6d9z" event={"ID":"bbb54d63-c3cc-4759-ba36-252e1ffc4d18","Type":"ContainerStarted","Data":"88f7490ae2fbb52873e01de2aa61d45b5d11f1f22e6ec8849f9a9d432763195e"}
Oct 30 12:35:43.458458 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.458419    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-84tjf" event={"ID":"e4fe0229-c475-400a-b6c0-48d80476b3b5","Type":"ContainerStarted","Data":"42d7c5b6e1a1b85ce656693813ea8f7eb392ce931e9a4e06a49a0d75bbefbfd8"}
Oct 30 12:35:43.462778 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.462739    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jsgw4" event={"ID":"eb31cffb-32c0-4d28-9947-325621908909","Type":"ContainerStarted","Data":"1e62a9e2b279520394ec02598bf1b5d4693b1355717d23bb9096425e1e53fbc8"}
Oct 30 12:35:43.465232 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.465202    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv" event={"ID":"0a195064-4c82-47ec-939e-8f0ac45fb0f6","Type":"ContainerStarted","Data":"c1699a8cb635ca88e61742e21af223617c6801b8f9d66639db3e0a89af6b497b"}
Oct 30 12:35:43.491877 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.491794    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6hvr" event={"ID":"996125c2-4148-45f2-845d-33e743fdd73a","Type":"ContainerStarted","Data":"a98e62006387c24317a806ce2becb38889c5ee81d1d2c62af2f8a74ff8d0bc14"}
Oct 30 12:35:43.640739 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.640701    2587 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:43.721854 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.721824    2587 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:43.933238 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.933157    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:43.933238 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:43.933207    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:43.933536 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:43.933285    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:43.933536 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:43.933352    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:43.933536 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:43.933359    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret podName:0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:45.93333976 +0000 UTC m=+5.124228032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret") pod "global-pull-secret-syncer-kd88h" (UID: "0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:43.933536 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:43.933422    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs podName:9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b nodeName:}" failed. No retries permitted until 2026-10-30 12:35:45.933386864 +0000 UTC m=+5.124275146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs") pod "network-metrics-daemon-v4ttl" (UID: "9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:44.034623 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:44.034573    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhm4h\" (UniqueName: \"kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h\") pod \"network-check-target-chqf2\" (UID: \"cd722809-68b9-41b8-8a5e-814903a74299\") " pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:44.034783 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:44.034749    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:35:44.034783 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:44.034775    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:35:44.034848 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:44.034788    2587 projected.go:196] Error preparing data for projected volume kube-api-access-fhm4h for pod openshift-network-diagnostics/network-check-target-chqf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:44.034883 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:44.034848    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h podName:cd722809-68b9-41b8-8a5e-814903a74299 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:46.034829194 +0000 UTC m=+5.225717465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fhm4h" (UniqueName: "kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h") pod "network-check-target-chqf2" (UID: "cd722809-68b9-41b8-8a5e-814903a74299") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:44.327762 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:44.327665    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:30:42 +0000 UTC" deadline="2028-08-13 21:12:30.436190738 +0000 UTC"
Oct 30 12:35:44.327762 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:44.327712    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15680h36m46.108484144s"
Oct 30 12:35:44.359861 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:44.359824    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:44.360029 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:44.359977    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:35:44.360616 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:44.360440    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:44.360616 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:44.360551    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:35:44.360767 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:44.360619    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:44.360767 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:44.360695    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:35:45.954298 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:45.954207    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:45.954298 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:45.954270    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:45.955020 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:45.954515    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:45.955020 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:45.954583    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs podName:9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b nodeName:}" failed. No retries permitted until 2026-10-30 12:35:49.954564181 +0000 UTC m=+9.145452452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs") pod "network-metrics-daemon-v4ttl" (UID: "9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:45.955152 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:45.955036    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:45.955152 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:45.955088    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret podName:0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:49.955072887 +0000 UTC m=+9.145961159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret") pod "global-pull-secret-syncer-kd88h" (UID: "0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:46.056202 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:46.055491    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhm4h\" (UniqueName: \"kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h\") pod \"network-check-target-chqf2\" (UID: \"cd722809-68b9-41b8-8a5e-814903a74299\") " pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:46.056202 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:46.055696    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:35:46.056202 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:46.055719    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:35:46.056202 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:46.055733    2587 projected.go:196] Error preparing data for projected volume kube-api-access-fhm4h for pod openshift-network-diagnostics/network-check-target-chqf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:46.056202 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:46.055801    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h podName:cd722809-68b9-41b8-8a5e-814903a74299 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:50.055780329 +0000 UTC m=+9.246668617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fhm4h" (UniqueName: "kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h") pod "network-check-target-chqf2" (UID: "cd722809-68b9-41b8-8a5e-814903a74299") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:46.359811 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:46.359775    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:46.359981 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:46.359927    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:35:46.360359 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:46.360337    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:46.360486 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:46.360465    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:35:46.360559 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:46.360539    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:46.360628 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:46.360609    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:35:46.513535 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:46.513280    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6hvr" event={"ID":"996125c2-4148-45f2-845d-33e743fdd73a","Type":"ContainerStarted","Data":"0eb65152424c6529d9bed8acaf3c86f17b04556d35778eb32484bba78ab21a86"}
Oct 30 12:35:47.519904 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:47.519549    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal" event={"ID":"72d3c22f1a6da7d8f3667e690d2cfffa","Type":"ContainerStarted","Data":"9bbf4a84f3c357de558a421f4a14ac46018f64120049285e6dbfb05de1aeea0d"}
Oct 30 12:35:47.524120 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:47.523383    2587 generic.go:356] "Generic (PLEG): container finished" podID="996125c2-4148-45f2-845d-33e743fdd73a" containerID="0eb65152424c6529d9bed8acaf3c86f17b04556d35778eb32484bba78ab21a86" exitCode=0
Oct 30 12:35:47.524120 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:47.523499    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6hvr" event={"ID":"996125c2-4148-45f2-845d-33e743fdd73a","Type":"ContainerDied","Data":"0eb65152424c6529d9bed8acaf3c86f17b04556d35778eb32484bba78ab21a86"}
Oct 30 12:35:47.529762 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:47.529729    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n6d9z" event={"ID":"bbb54d63-c3cc-4759-ba36-252e1ffc4d18","Type":"ContainerStarted","Data":"1121aacc328e05f590e91552733040a9120bcea8ebacabaa54a22acfbef26b6b"}
Oct 30 12:35:47.533868 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:47.533281    2587 generic.go:356] "Generic (PLEG): container finished" podID="6068dd25b70d52ed49d0937c3f3549a5" containerID="f15724b0f674456c003f00775aa7ed554b92566acd2a36acbf5989bcccdf14cc" exitCode=0
Oct 30 12:35:47.533868 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:47.533327    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal" event={"ID":"6068dd25b70d52ed49d0937c3f3549a5","Type":"ContainerDied","Data":"f15724b0f674456c003f00775aa7ed554b92566acd2a36acbf5989bcccdf14cc"}
Oct 30 12:35:47.597677 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:47.597605    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal" podStartSLOduration=6.597585828 podStartE2EDuration="6.597585828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:47.556553078 +0000 UTC m=+6.747441361" watchObservedRunningTime="2026-10-30 12:35:47.597585828 +0000 UTC m=+6.788474107"
Oct 30 12:35:47.639003 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:47.638615    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n6d9z" podStartSLOduration=2.523239834 podStartE2EDuration="6.638594492s" totalImagesPullingTime="4.115354658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:47.59879373 +0000 UTC m=+6.789682011" watchObservedRunningTime="2026-10-30 12:35:47.638594492 +0000 UTC m=+6.829482772"
Oct 30 12:35:48.360228 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:48.360193    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:48.360407 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:48.360339    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:35:48.360824 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:48.360800    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:48.360940 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:48.360919    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:35:48.361013 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:48.360985    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:48.361074 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:48.361055    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:35:48.541577 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:48.540885    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p9gs7" event={"ID":"c72d4793-688e-4c37-bb51-e0a9100a2203","Type":"ContainerStarted","Data":"0f3bd3efc6bc9344f46944525a968e950e301dafb1148d96f760beae75c1388a"}
Oct 30 12:35:48.547978 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:48.547564    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal" event={"ID":"6068dd25b70d52ed49d0937c3f3549a5","Type":"ContainerStarted","Data":"9d6a5111481351fa18152f5aa1b847e1f9456f2cb9f91e8bd911ed8f5dce39d7"}
Oct 30 12:35:48.616170 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:48.615107    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p9gs7" podStartSLOduration=2.05449338 podStartE2EDuration="7.615091424s" totalImagesPullingTime="5.560598044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:48.578878357 +0000 UTC m=+7.769766638" watchObservedRunningTime="2026-10-30 12:35:48.615091424 +0000 UTC m=+7.805979703"
Oct 30 12:35:49.264247 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:49.264106    2587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:35:49.264636 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:49.264577    2587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:35:49.300998 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:49.300935    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal" podStartSLOduration=8.300912167 podStartE2EDuration="8.300912167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:48.616143953 +0000 UTC m=+7.807032214" watchObservedRunningTime="2026-10-30 12:35:49.300912167 +0000 UTC m=+8.491800450"
Oct 30 12:35:49.560245 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:49.560155    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v" event={"ID":"ca860770-e681-4dae-806f-ef6de17bed8e","Type":"ContainerStarted","Data":"2effdb7a644ddd8f537cd21ef4efe983ecfe4f1f6c7013f9407d895f55d7b035"}
Oct 30 12:35:49.994478 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:49.994441    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:49.994673 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:49.994494    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:49.994673 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:49.994653    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:49.994792 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:49.994703    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs podName:9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b nodeName:}" failed. No retries permitted until 2026-10-30 12:35:57.994689953 +0000 UTC m=+17.185578223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs") pod "network-metrics-daemon-v4ttl" (UID: "9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:49.995096 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:49.995076    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:49.995175 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:49.995129    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret podName:0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:57.995114307 +0000 UTC m=+17.186002579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret") pod "global-pull-secret-syncer-kd88h" (UID: "0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:50.095737 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:50.095697    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhm4h\" (UniqueName: \"kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h\") pod \"network-check-target-chqf2\" (UID: \"cd722809-68b9-41b8-8a5e-814903a74299\") " pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:50.095988 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:50.095875    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:35:50.095988 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:50.095898    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:35:50.095988 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:50.095911    2587 projected.go:196] Error preparing data for projected volume kube-api-access-fhm4h for pod openshift-network-diagnostics/network-check-target-chqf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:50.095988 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:50.095971    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h podName:cd722809-68b9-41b8-8a5e-814903a74299 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:58.095951469 +0000 UTC m=+17.286839740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fhm4h" (UniqueName: "kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h") pod "network-check-target-chqf2" (UID: "cd722809-68b9-41b8-8a5e-814903a74299") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:50.360068 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:50.359985    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:50.360222 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:50.360135    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:35:50.360604 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:50.360578    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:50.360719 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:50.360685    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:35:50.360783 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:50.360743    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:50.360836 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:50.360810    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:35:50.562969 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:50.562937    2587 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:35:50.707857 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:50.707762    2587 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 12:35:51.279415 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:51.279291    2587 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:35:50.707780672Z","UUID":"92d26c2b-3810-45f9-a5a1-d328f4406103","Handler":null,"Name":"","Endpoint":""}
Oct 30 12:35:51.281704 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:51.281223    2587 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 12:35:51.281704 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:51.281273    2587 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 12:35:51.573180 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:51.573090    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-84tjf" event={"ID":"e4fe0229-c475-400a-b6c0-48d80476b3b5","Type":"ContainerStarted","Data":"f23a39b2b3c3d64aabbfc5be59738c30e127b67e8c9ca7b26e1995e33d576047"}
Oct 30 12:35:51.577827 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:51.577784    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6hvr" event={"ID":"996125c2-4148-45f2-845d-33e743fdd73a","Type":"ContainerStarted","Data":"8d9e647ffdc3d281d2d984ba8247d017bdb03baa35ed34d350ba06caa671b763"}
Oct 30 12:35:51.580995 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:51.580607    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5dfvg" event={"ID":"14b22d89-19e4-4dbb-a031-91b5fe337785","Type":"ContainerStarted","Data":"bf893c1bc761d2518fe70b118532a173977c0bad69a7628aa98220d3df0a09e7"}
Oct 30 12:35:51.585189 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:51.585153    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v" event={"ID":"ca860770-e681-4dae-806f-ef6de17bed8e","Type":"ContainerStarted","Data":"49994c1ab145d7a4b94ad879f10077f63ec183abc7aaf0ecb6a5f960560f5b92"}
Oct 30 12:35:51.652727 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:51.649738    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-5dfvg" podStartSLOduration=2.3486752109999998 podStartE2EDuration="10.649719458s" totalImagesPullingTime="8.301044247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:51.649148932 +0000 UTC m=+10.840037202" watchObservedRunningTime="2026-10-30 12:35:51.649719458 +0000 UTC m=+10.840607740"
Oct 30 12:35:51.652727 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:51.650670    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-84tjf" podStartSLOduration=2.01150104 podStartE2EDuration="10.650660184s" totalImagesPullingTime="8.639159144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:51.60999462 +0000 UTC m=+10.800882903" watchObservedRunningTime="2026-10-30 12:35:51.650660184 +0000 UTC m=+10.841548463"
Oct 30 12:35:52.360043 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:52.360002    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:52.360043 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:52.360048    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:52.360246 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:52.360011    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:52.360246 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:52.360149    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:35:52.360348 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:52.360249    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:35:52.360424 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:52.360372    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:35:52.590804 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:52.590765    2587 generic.go:356] "Generic (PLEG): container finished" podID="996125c2-4148-45f2-845d-33e743fdd73a" containerID="8d9e647ffdc3d281d2d984ba8247d017bdb03baa35ed34d350ba06caa671b763" exitCode=0
Oct 30 12:35:52.591237 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:52.590859    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6hvr" event={"ID":"996125c2-4148-45f2-845d-33e743fdd73a","Type":"ContainerDied","Data":"8d9e647ffdc3d281d2d984ba8247d017bdb03baa35ed34d350ba06caa671b763"}
Oct 30 12:35:52.594132 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:52.594103    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vmqth" event={"ID":"233c3bd3-01bc-4c71-8bc4-611933e34539","Type":"ContainerStarted","Data":"255623c3a8140fad9e7dbd1c0dcbfd70ce59414cf4b09e26f8f8f2f8f464583f"}
Oct 30 12:35:52.597862 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:52.597832    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v" event={"ID":"ca860770-e681-4dae-806f-ef6de17bed8e","Type":"ContainerStarted","Data":"fd288915c554bd2f60d2599b3842922829aed5cd7b87237cdfefbfc31cf74d47"}
Oct 30 12:35:52.675922 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:52.675811    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v" podStartSLOduration=2.756250722 podStartE2EDuration="11.675790889s" totalImagesPullingTime="8.919540167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:52.675638212 +0000 UTC m=+11.866526491" watchObservedRunningTime="2026-10-30 12:35:52.675790889 +0000 UTC m=+11.866679170"
Oct 30 12:35:52.719418 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:52.719346    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vmqth" podStartSLOduration=3.31976879 podStartE2EDuration="11.719334032s" totalImagesPullingTime="8.399565242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:52.71893691 +0000 UTC m=+11.909825191" watchObservedRunningTime="2026-10-30 12:35:52.719334032 +0000 UTC m=+11.910222311"
Oct 30 12:35:53.602383 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:53.602165    2587 generic.go:356] "Generic (PLEG): container finished" podID="996125c2-4148-45f2-845d-33e743fdd73a" containerID="64d6b8248ca7de6cd08a8f6db1cd8cadc55bf2c3216bcfe5b538d4056b5832cc" exitCode=0
Oct 30 12:35:53.602877 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:53.602264    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6hvr" event={"ID":"996125c2-4148-45f2-845d-33e743fdd73a","Type":"ContainerDied","Data":"64d6b8248ca7de6cd08a8f6db1cd8cadc55bf2c3216bcfe5b538d4056b5832cc"}
Oct 30 12:35:53.605432 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:53.605410    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jsgw4" event={"ID":"eb31cffb-32c0-4d28-9947-325621908909","Type":"ContainerStarted","Data":"90d26b4b09353819c778da02323dbbbba76ffa04306f87c95ebc77f66ce4878b"}
Oct 30 12:35:53.693268 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:53.693202    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jsgw4" podStartSLOduration=2.136069539 podStartE2EDuration="12.69318207s" totalImagesPullingTime="10.557112531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:53.692440685 +0000 UTC m=+12.883328963" watchObservedRunningTime="2026-10-30 12:35:53.69318207 +0000 UTC m=+12.884070350"
Oct 30 12:35:54.360544 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:54.360501    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:54.360763 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:54.360593    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:54.360763 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:54.360702    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:54.360763 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:54.360714    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:35:54.360928 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:54.360808    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:35:54.360928 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:54.360909    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:35:54.610236 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:54.610201    2587 generic.go:356] "Generic (PLEG): container finished" podID="996125c2-4148-45f2-845d-33e743fdd73a" containerID="ba987709f8e53fb5a0674615363b6b01f666e57b6016c6cf7329c1faefd628da" exitCode=0
Oct 30 12:35:54.610820 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:54.610259    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6hvr" event={"ID":"996125c2-4148-45f2-845d-33e743fdd73a","Type":"ContainerDied","Data":"ba987709f8e53fb5a0674615363b6b01f666e57b6016c6cf7329c1faefd628da"}
Oct 30 12:35:56.360565 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:56.360209    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:56.360565 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:56.360237    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:56.360565 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:56.360241    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:56.360565 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:56.360349    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:35:56.360565 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:56.360459    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:35:56.360565 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:56.360536    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:35:56.619325 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:56.619240    2587 generic.go:356] "Generic (PLEG): container finished" podID="996125c2-4148-45f2-845d-33e743fdd73a" containerID="6c8a9fd4d969002b4cbf2e687d767e2a0fd1552a6e0f53436d78621ec0020648" exitCode=0
Oct 30 12:35:56.619325 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:56.619289    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6hvr" event={"ID":"996125c2-4148-45f2-845d-33e743fdd73a","Type":"ContainerDied","Data":"6c8a9fd4d969002b4cbf2e687d767e2a0fd1552a6e0f53436d78621ec0020648"}
Oct 30 12:35:57.624694 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:57.624494    2587 generic.go:356] "Generic (PLEG): container finished" podID="996125c2-4148-45f2-845d-33e743fdd73a" containerID="b92aa2f0a01ccf9b3e141af86eb31c0cab38c43aebaef73e0a617f7e07879497" exitCode=0
Oct 30 12:35:57.624694 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:57.624567    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6hvr" event={"ID":"996125c2-4148-45f2-845d-33e743fdd73a","Type":"ContainerDied","Data":"b92aa2f0a01ccf9b3e141af86eb31c0cab38c43aebaef73e0a617f7e07879497"}
Oct 30 12:35:57.810952 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:57.810927    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-84tjf_e4fe0229-c475-400a-b6c0-48d80476b3b5/dns-node-resolver/0.log"
Oct 30 12:35:58.062118 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.062091    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:58.062231 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.062125    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:58.062264 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:58.062227    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:58.062264 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:58.062233    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:58.062324 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:58.062284    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret podName:0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4 nodeName:}" failed. No retries permitted until 2026-10-30 12:36:14.062269185 +0000 UTC m=+33.253157441 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret") pod "global-pull-secret-syncer-kd88h" (UID: "0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:58.062324 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:58.062297    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs podName:9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b nodeName:}" failed. No retries permitted until 2026-10-30 12:36:14.062290682 +0000 UTC m=+33.253178939 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs") pod "network-metrics-daemon-v4ttl" (UID: "9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:58.162901 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.162821    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhm4h\" (UniqueName: \"kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h\") pod \"network-check-target-chqf2\" (UID: \"cd722809-68b9-41b8-8a5e-814903a74299\") " pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:58.163030 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:58.162971    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:35:58.163030 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:58.162988    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:35:58.163030 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:58.162997    2587 projected.go:196] Error preparing data for projected volume kube-api-access-fhm4h for pod openshift-network-diagnostics/network-check-target-chqf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:58.163121 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:58.163047    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h podName:cd722809-68b9-41b8-8a5e-814903a74299 nodeName:}" failed. No retries permitted until 2026-10-30 12:36:14.163033091 +0000 UTC m=+33.353921349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fhm4h" (UniqueName: "kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h") pod "network-check-target-chqf2" (UID: "cd722809-68b9-41b8-8a5e-814903a74299") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:58.360468 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.360432    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:35:58.360663 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.360476    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:35:58.360663 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.360566    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:35:58.360663 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:58.360579    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:35:58.360820 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:58.360685    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:35:58.360820 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:35:58.360773    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:35:58.629929 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.629901    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 12:35:58.630466 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.630255    2587 generic.go:356] "Generic (PLEG): container finished" podID="0a195064-4c82-47ec-939e-8f0ac45fb0f6" containerID="07db58e1fe193b2ef936f10833c14323497196bde80555fa41f6e550f9e9a5c1" exitCode=1
Oct 30 12:35:58.630466 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.630322    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv" event={"ID":"0a195064-4c82-47ec-939e-8f0ac45fb0f6","Type":"ContainerStarted","Data":"4d409b832c27f9ac9fc42ab2abe50e0ffadfc549c9605f061e4decc9863703d8"}
Oct 30 12:35:58.630466 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.630438    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv" event={"ID":"0a195064-4c82-47ec-939e-8f0ac45fb0f6","Type":"ContainerStarted","Data":"6b4ac135bc5184979637f8a91101a52fd36972e26e35b042aa6bf3ecf1bcb54b"}
Oct 30 12:35:58.630590 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.630492    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv" event={"ID":"0a195064-4c82-47ec-939e-8f0ac45fb0f6","Type":"ContainerStarted","Data":"8cccbce59326c60e9b75497dec82a9f0673308a8123ce62db7a03d43a3a11c22"}
Oct 30 12:35:58.630590 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.630540    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv" event={"ID":"0a195064-4c82-47ec-939e-8f0ac45fb0f6","Type":"ContainerStarted","Data":"d82c8b2d38ca059cbd5b46a23557807b89c98eb97e40345aa4d6bdcd956b03a4"}
Oct 30 12:35:58.630590 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.630572    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv" event={"ID":"0a195064-4c82-47ec-939e-8f0ac45fb0f6","Type":"ContainerDied","Data":"07db58e1fe193b2ef936f10833c14323497196bde80555fa41f6e550f9e9a5c1"}
Oct 30 12:35:58.630704 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.630601    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv" event={"ID":"0a195064-4c82-47ec-939e-8f0ac45fb0f6","Type":"ContainerStarted","Data":"7a223f91c61ab577570c9817d845b30226a69eb696ad75357a0e35746cbcf740"}
Oct 30 12:35:58.633693 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.633672    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6hvr" event={"ID":"996125c2-4148-45f2-845d-33e743fdd73a","Type":"ContainerStarted","Data":"c66cafe3c23ffca23a8c57174c042eec9094db99af741a906bbfb7d3010d1f10"}
Oct 30 12:35:58.764258 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:35:58.764089    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p9gs7_c72d4793-688e-4c37-bb51-e0a9100a2203/node-ca/0.log"
Oct 30 12:36:00.360339 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:00.360287    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:00.360339 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:00.360329    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:00.360866 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:00.360380    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:00.360866 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:00.360412    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:36:00.360866 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:00.360479    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:36:00.360866 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:00.360547    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:36:00.640563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:00.640537    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 12:36:00.640896 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:00.640873    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv" event={"ID":"0a195064-4c82-47ec-939e-8f0ac45fb0f6","Type":"ContainerStarted","Data":"54b984997af4b6154dea55b40bac2cba8aa9738d8465cf577c566cf3c92ddb40"}
Oct 30 12:36:02.360364 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:02.360340    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:02.360878 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:02.360387    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:02.360878 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:02.360494    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:36:02.360878 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:02.360556    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:36:02.360878 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:02.360573    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:02.360878 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:02.360638    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:36:02.648624 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:02.648546    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 12:36:02.648900 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:02.648878    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv" event={"ID":"0a195064-4c82-47ec-939e-8f0ac45fb0f6","Type":"ContainerStarted","Data":"0f6faa428a8bec1e6e2273474aef81ed9a13b956dc98d770174270b6fee65895"}
Oct 30 12:36:02.649162 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:02.649141    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:02.649314 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:02.649295    2587 scope.go:122] "RemoveContainer" containerID="07db58e1fe193b2ef936f10833c14323497196bde80555fa41f6e550f9e9a5c1"
Oct 30 12:36:02.666213 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:02.666181    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:02.708039 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:02.707987    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x6hvr" podStartSLOduration=11.969748241 podStartE2EDuration="21.707969792s" totalImagesPullingTime="9.738221551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:58.677043391 +0000 UTC m=+17.867931682" watchObservedRunningTime="2026-10-30 12:36:02.707969792 +0000 UTC m=+21.898858072"
Oct 30 12:36:02.819023 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:02.818999    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:36:02.819222 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:02.819119    2587 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:36:02.819568 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:02.819553    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:36:03.654960 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:03.654927    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 12:36:03.655887 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:03.655267    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv" event={"ID":"0a195064-4c82-47ec-939e-8f0ac45fb0f6","Type":"ContainerStarted","Data":"67865e22dfaea76c5a6929e0fb0ca3d44d209bcb72523ecd2f486b0934dbbe02"}
Oct 30 12:36:03.655887 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:03.655416    2587 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:36:03.655887 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:03.655735    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:03.671686 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:03.671662    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:03.756448 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:03.756372    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv" podStartSLOduration=7.6449778219999995 podStartE2EDuration="22.756353756s" totalImagesPullingTime="15.111375934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:03.755841855 +0000 UTC m=+22.946730127" watchObservedRunningTime="2026-10-30 12:36:03.756353756 +0000 UTC m=+22.947242035"
Oct 30 12:36:04.360360 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:04.360323    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:04.360554 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:04.360331    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:04.360554 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:04.360338    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:04.360554 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:04.360495    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:36:04.360672 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:04.360568    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:36:04.360672 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:04.360625    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:36:04.657717 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:04.657641    2587 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:36:05.659553 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:05.659523    2587 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:36:06.360550 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:06.360505    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:06.360550 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:06.360536    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:06.360550 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:06.360546    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:06.360785 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:06.360622    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:36:06.360785 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:06.360679    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:36:06.360785 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:06.360758    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:36:08.359607 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:08.359576    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:08.360077 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:08.359686    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:08.360077 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:08.359779    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:36:08.360077 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:08.359691    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:36:08.360077 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:08.359692    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:08.360077 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:08.359858    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:36:09.304254 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:09.304220    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:09.304488 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:09.304475    2587 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:36:09.314706 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:09.314667    2587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv" podUID="0a195064-4c82-47ec-939e-8f0ac45fb0f6" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 12:36:09.323871 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:09.323842    2587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv" podUID="0a195064-4c82-47ec-939e-8f0ac45fb0f6" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 12:36:10.360492 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:10.360460    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:10.360492 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:10.360506    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:10.360978 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:10.360522    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:10.360978 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:10.360593    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:36:10.360978 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:10.360631    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:36:10.360978 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:10.360744    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:36:12.360651 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:12.360431    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:12.361097 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:12.360453    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:12.361097 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:12.360743    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:36:12.361097 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:12.360819    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:36:12.361097 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:12.360465    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:12.361097 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:12.360914    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:36:14.067204 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:14.067165    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:14.067204 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:14.067209    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:14.067672 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:14.067339    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:36:14.067672 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:14.067428    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret podName:0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4 nodeName:}" failed. No retries permitted until 2026-10-30 12:36:46.067382912 +0000 UTC m=+65.258271169 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret") pod "global-pull-secret-syncer-kd88h" (UID: "0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:36:14.067672 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:14.067339    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:36:14.067672 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:14.067474    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs podName:9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b nodeName:}" failed. No retries permitted until 2026-10-30 12:36:46.067462428 +0000 UTC m=+65.258350689 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs") pod "network-metrics-daemon-v4ttl" (UID: "9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:36:14.168557 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:14.168519    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhm4h\" (UniqueName: \"kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h\") pod \"network-check-target-chqf2\" (UID: \"cd722809-68b9-41b8-8a5e-814903a74299\") " pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:14.168721 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:14.168641    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:36:14.168721 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:14.168658    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:36:14.168721 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:14.168668    2587 projected.go:196] Error preparing data for projected volume kube-api-access-fhm4h for pod openshift-network-diagnostics/network-check-target-chqf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:36:14.168721 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:14.168719    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h podName:cd722809-68b9-41b8-8a5e-814903a74299 nodeName:}" failed. No retries permitted until 2026-10-30 12:36:46.168706604 +0000 UTC m=+65.359594861 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fhm4h" (UniqueName: "kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h") pod "network-check-target-chqf2" (UID: "cd722809-68b9-41b8-8a5e-814903a74299") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:36:14.360456 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:14.360352    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:14.360598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:14.360455    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:14.360598 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:14.360548    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:14.360598 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:14.360557    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:36:14.360701 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:14.360616    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:36:14.360753 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:14.360727    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:36:16.360494 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:16.360461    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:16.360956 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:16.360496    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:16.360956 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:16.360589    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:36:16.360956 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:16.360641    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:36:16.360956 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:16.360669    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:16.360956 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:16.360777    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:36:16.460024 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:16.459939    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kd88h"]
Oct 30 12:36:16.463614 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:16.463335    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v4ttl"]
Oct 30 12:36:16.463913 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:16.463893    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-chqf2"]
Oct 30 12:36:16.685103 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:16.685077    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:16.685300 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:16.685083    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:16.685300 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:16.685173    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:36:16.685300 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:16.685083    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:16.685300 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:16.685268    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:36:16.685492 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:16.685328    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:36:18.360066 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:18.360035    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:18.360066 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:18.360059    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:18.360583 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:18.360036    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:18.360583 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:18.360145    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:36:18.360583 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:18.360228    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:36:18.360583 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:18.360310    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:36:20.360165 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.360132    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:20.360165 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.360151    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:20.360574 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.360182    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:20.360574 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:20.360273    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4ttl" podUID="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b"
Oct 30 12:36:20.360574 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:20.360379    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-chqf2" podUID="cd722809-68b9-41b8-8a5e-814903a74299"
Oct 30 12:36:20.360574 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:20.360508    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd88h" podUID="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4"
Oct 30 12:36:20.594382 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.594342    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-228.ec2.internal" event="NodeReady"
Oct 30 12:36:20.594554 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.594505    2587 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 12:36:20.655558 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.655474    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zh8sx"]
Oct 30 12:36:20.679221 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.679179    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dg6tk"]
Oct 30 12:36:20.679427 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.679320    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:20.692711 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.692684    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zh8sx"]
Oct 30 12:36:20.692844 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.692729    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dg6tk"]
Oct 30 12:36:20.692844 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.692804    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.693470 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.693452    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 12:36:20.693588 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.693551    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tz7hb\""
Oct 30 12:36:20.693660 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.693624    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 12:36:20.706576 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.706552    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 12:36:20.706699 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.706598    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 12:36:20.706699 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.706624    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 12:36:20.706699 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.706643    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 12:36:20.706804 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.706711    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t86w7\""
Oct 30 12:36:20.768073 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.768035    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qmqdw"]
Oct 30 12:36:20.783050 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.783017    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qmqdw"
Oct 30 12:36:20.799409 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.799369    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pdckv\""
Oct 30 12:36:20.804627 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.804602    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 12:36:20.805045 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.805026    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 12:36:20.805306 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.805283    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qmqdw"]
Oct 30 12:36:20.816254 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.816229    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e383bb23-4967-47e8-afdf-937f840aea60-data-volume\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.816420 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.816261    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2343d79d-4146-4f1f-90be-ad84a256a4a5-metrics-tls\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:20.816420 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.816280    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e383bb23-4967-47e8-afdf-937f840aea60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.816503 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.816450    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbhd4\" (UniqueName: \"kubernetes.io/projected/2343d79d-4146-4f1f-90be-ad84a256a4a5-kube-api-access-tbhd4\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:20.816503 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.816478    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e383bb23-4967-47e8-afdf-937f840aea60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.816572 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.816548    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2343d79d-4146-4f1f-90be-ad84a256a4a5-config-volume\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:20.816572 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.816565    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2343d79d-4146-4f1f-90be-ad84a256a4a5-tmp-dir\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:20.816643 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.816585    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e383bb23-4967-47e8-afdf-937f840aea60-crio-socket\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.816643 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.816621    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkqw9\" (UniqueName: \"kubernetes.io/projected/e383bb23-4967-47e8-afdf-937f840aea60-kube-api-access-nkqw9\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.899053 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.899019    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 12:36:20.917283 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.917217    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ddd38ac5-1413-4345-9452-05936c2b31b8-cert\") pod \"ingress-canary-qmqdw\" (UID: \"ddd38ac5-1413-4345-9452-05936c2b31b8\") " pod="openshift-ingress-canary/ingress-canary-qmqdw"
Oct 30 12:36:20.917283 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.917258    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2343d79d-4146-4f1f-90be-ad84a256a4a5-metrics-tls\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:20.917283 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.917279    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e383bb23-4967-47e8-afdf-937f840aea60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.917514 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.917489    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbhd4\" (UniqueName: \"kubernetes.io/projected/2343d79d-4146-4f1f-90be-ad84a256a4a5-kube-api-access-tbhd4\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:20.917561 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.917530    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e383bb23-4967-47e8-afdf-937f840aea60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.917615 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.917563    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2343d79d-4146-4f1f-90be-ad84a256a4a5-config-volume\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:20.917615 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.917587    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2343d79d-4146-4f1f-90be-ad84a256a4a5-tmp-dir\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:20.917703 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.917618    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e383bb23-4967-47e8-afdf-937f840aea60-crio-socket\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.917703 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.917659    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkqw9\" (UniqueName: \"kubernetes.io/projected/e383bb23-4967-47e8-afdf-937f840aea60-kube-api-access-nkqw9\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.917792 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.917703    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e383bb23-4967-47e8-afdf-937f840aea60-data-volume\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.917792 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.917728    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndn6b\" (UniqueName: \"kubernetes.io/projected/ddd38ac5-1413-4345-9452-05936c2b31b8-kube-api-access-ndn6b\") pod \"ingress-canary-qmqdw\" (UID: \"ddd38ac5-1413-4345-9452-05936c2b31b8\") " pod="openshift-ingress-canary/ingress-canary-qmqdw"
Oct 30 12:36:20.917880 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.917849    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e383bb23-4967-47e8-afdf-937f840aea60-crio-socket\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.918146 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.918124    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e383bb23-4967-47e8-afdf-937f840aea60-data-volume\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.918243 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.918221    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e383bb23-4967-47e8-afdf-937f840aea60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.918290 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.918221    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2343d79d-4146-4f1f-90be-ad84a256a4a5-config-volume\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:20.918290 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.918272    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2343d79d-4146-4f1f-90be-ad84a256a4a5-tmp-dir\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:20.921158 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.921136    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e383bb23-4967-47e8-afdf-937f840aea60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.921275 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.921142    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2343d79d-4146-4f1f-90be-ad84a256a4a5-metrics-tls\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:20.936823 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.936790    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkqw9\" (UniqueName: \"kubernetes.io/projected/e383bb23-4967-47e8-afdf-937f840aea60-kube-api-access-nkqw9\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:20.936931 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:20.936827    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbhd4\" (UniqueName: \"kubernetes.io/projected/2343d79d-4146-4f1f-90be-ad84a256a4a5-kube-api-access-tbhd4\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:21.011986 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:21.011948    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:21.018219 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:21.018187    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndn6b\" (UniqueName: \"kubernetes.io/projected/ddd38ac5-1413-4345-9452-05936c2b31b8-kube-api-access-ndn6b\") pod \"ingress-canary-qmqdw\" (UID: \"ddd38ac5-1413-4345-9452-05936c2b31b8\") " pod="openshift-ingress-canary/ingress-canary-qmqdw"
Oct 30 12:36:21.018301 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:21.018224    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ddd38ac5-1413-4345-9452-05936c2b31b8-cert\") pod \"ingress-canary-qmqdw\" (UID: \"ddd38ac5-1413-4345-9452-05936c2b31b8\") " pod="openshift-ingress-canary/ingress-canary-qmqdw"
Oct 30 12:36:21.020816 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:21.020790    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ddd38ac5-1413-4345-9452-05936c2b31b8-cert\") pod \"ingress-canary-qmqdw\" (UID: \"ddd38ac5-1413-4345-9452-05936c2b31b8\") " pod="openshift-ingress-canary/ingress-canary-qmqdw"
Oct 30 12:36:21.037057 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:21.037031    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndn6b\" (UniqueName: \"kubernetes.io/projected/ddd38ac5-1413-4345-9452-05936c2b31b8-kube-api-access-ndn6b\") pod \"ingress-canary-qmqdw\" (UID: \"ddd38ac5-1413-4345-9452-05936c2b31b8\") " pod="openshift-ingress-canary/ingress-canary-qmqdw"
Oct 30 12:36:21.053596 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:21.053567    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:21.106023 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:21.105758    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qmqdw"
Oct 30 12:36:21.416295 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:21.416274    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zh8sx"]
Oct 30 12:36:21.419047 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:36:21.419027    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2343d79d_4146_4f1f_90be_ad84a256a4a5.slice/crio-5d4d1df1301def3a46aa74074625f528eebf276bef4a26e3e8cce05397f6189b WatchSource:0}: Error finding container 5d4d1df1301def3a46aa74074625f528eebf276bef4a26e3e8cce05397f6189b: Status 404 returned error can't find the container with id 5d4d1df1301def3a46aa74074625f528eebf276bef4a26e3e8cce05397f6189b
Oct 30 12:36:21.419955 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:21.419923    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dg6tk"]
Oct 30 12:36:21.421067 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:36:21.421044    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode383bb23_4967_47e8_afdf_937f840aea60.slice/crio-45356e460e3e75b3bb9e7bd3d0049c5c050a496174f25db8d7bd3791152318e1 WatchSource:0}: Error finding container 45356e460e3e75b3bb9e7bd3d0049c5c050a496174f25db8d7bd3791152318e1: Status 404 returned error can't find the container with id 45356e460e3e75b3bb9e7bd3d0049c5c050a496174f25db8d7bd3791152318e1
Oct 30 12:36:21.426232 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:21.426211    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qmqdw"]
Oct 30 12:36:21.429027 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:36:21.429005    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd38ac5_1413_4345_9452_05936c2b31b8.slice/crio-0855e4c7631992f259706fb48208b75d9b90d1740b84731caa06f74f63646877 WatchSource:0}: Error finding container 0855e4c7631992f259706fb48208b75d9b90d1740b84731caa06f74f63646877: Status 404 returned error can't find the container with id 0855e4c7631992f259706fb48208b75d9b90d1740b84731caa06f74f63646877
Oct 30 12:36:21.698083 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:21.697994    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qmqdw" event={"ID":"ddd38ac5-1413-4345-9452-05936c2b31b8","Type":"ContainerStarted","Data":"0855e4c7631992f259706fb48208b75d9b90d1740b84731caa06f74f63646877"}
Oct 30 12:36:21.699452 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:21.699389    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dg6tk" event={"ID":"e383bb23-4967-47e8-afdf-937f840aea60","Type":"ContainerStarted","Data":"c1d01c44515e82d6b55ec9f1e2ce8a390c91b482ddb75b99ffe3f02d4d01b2b0"}
Oct 30 12:36:21.699583 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:21.699474    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dg6tk" event={"ID":"e383bb23-4967-47e8-afdf-937f840aea60","Type":"ContainerStarted","Data":"45356e460e3e75b3bb9e7bd3d0049c5c050a496174f25db8d7bd3791152318e1"}
Oct 30 12:36:21.700525 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:21.700502    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zh8sx" event={"ID":"2343d79d-4146-4f1f-90be-ad84a256a4a5","Type":"ContainerStarted","Data":"5d4d1df1301def3a46aa74074625f528eebf276bef4a26e3e8cce05397f6189b"}
Oct 30 12:36:22.359770 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:22.359731    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:22.359770 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:22.359765    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:22.360007 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:22.359771    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:22.374338 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:22.374277    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:36:22.386109 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:22.386079    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bg2k7\""
Oct 30 12:36:22.386248 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:22.386158    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-t6s7j\""
Oct 30 12:36:22.386248 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:22.386162    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:36:22.386248 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:22.386233    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:36:22.386454 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:22.386286    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-54vc4\""
Oct 30 12:36:22.386454 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:22.386368    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:36:23.599498 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.599467    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"]
Oct 30 12:36:23.618428 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.618381    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:23.621588 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.621559    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"]
Oct 30 12:36:23.635292 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.635092    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 12:36:23.635292 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.635165    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 12:36:23.635292 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.635209    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 12:36:23.647775 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.647741    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9gvsj"]
Oct 30 12:36:23.649435 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.649386    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 12:36:23.649586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.649509    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-whrjh\""
Oct 30 12:36:23.649586 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.649558    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 12:36:23.663240 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.663208    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"]
Oct 30 12:36:23.664322 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.664299    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.678077 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.677621    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"]
Oct 30 12:36:23.678077 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.677760    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.735205 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.735176    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 12:36:23.735355 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.735262    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 12:36:23.735355 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.735254    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 12:36:23.735355 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.735301    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-s26wd\""
Oct 30 12:36:23.735549 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.735387    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-87th4\""
Oct 30 12:36:23.735549 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.735543    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 12:36:23.735647 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.735596    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 12:36:23.735647 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.735627    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 12:36:23.742158 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.742126    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:23.742296 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.742175    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbg47\" (UniqueName: \"kubernetes.io/projected/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-kube-api-access-lbg47\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:23.742296 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.742285    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:23.742426 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.742326    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:23.746028 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.745997    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dg6tk" event={"ID":"e383bb23-4967-47e8-afdf-937f840aea60","Type":"ContainerStarted","Data":"94e0916a025bcc746b7b1384f2d16944415edfc84351258a1ed5499be52315a8"}
Oct 30 12:36:23.844563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.843773    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.844563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844089    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-textfile\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.844563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844122    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqhpx\" (UniqueName: \"kubernetes.io/projected/70b50531-ed83-4bac-bdf1-0992042e01a0-kube-api-access-xqhpx\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.844563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844156    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7ad82f0-86a1-43e3-8237-846a3aef0c88-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.844563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844195    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:23.844563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844222    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-tls\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.844563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844253    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:23.844563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844284    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.844563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844302    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc6np\" (UniqueName: \"kubernetes.io/projected/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-api-access-gc6np\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.844563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844321    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.844563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844338    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f7ad82f0-86a1-43e3-8237-846a3aef0c88-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.844563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844355    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-sys\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.844563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844369    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70b50531-ed83-4bac-bdf1-0992042e01a0-metrics-client-ca\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.844563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844422    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-root\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.844563 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844449    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-wtmp\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.845382 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844535    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.845382 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844737    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:23.845382 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.844814    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbg47\" (UniqueName: \"kubernetes.io/projected/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-kube-api-access-lbg47\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:23.845382 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.845204    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.845885 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.845765    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:23.848954 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.848700    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:23.848954 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.848854    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:23.869135 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.869105    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbg47\" (UniqueName: \"kubernetes.io/projected/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-kube-api-access-lbg47\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:23.946351 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.946318    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-textfile\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.946351 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.946359    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhpx\" (UniqueName: \"kubernetes.io/projected/70b50531-ed83-4bac-bdf1-0992042e01a0-kube-api-access-xqhpx\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.946622 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.946380    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7ad82f0-86a1-43e3-8237-846a3aef0c88-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.946622 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.946591    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-tls\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.946727 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.946646    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.946727 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.946678    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc6np\" (UniqueName: \"kubernetes.io/projected/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-api-access-gc6np\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.946727 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.946708    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.946864 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.946735    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f7ad82f0-86a1-43e3-8237-846a3aef0c88-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.946864 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.946761    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-sys\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.946864 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.946774    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-textfile\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.946864 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.946795    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70b50531-ed83-4bac-bdf1-0992042e01a0-metrics-client-ca\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.946864 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.946844    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-root\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.947091 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.946873    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-wtmp\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.947091 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.946905    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.947091 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.946948    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.947091 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.947005    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.947280 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.947203    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7ad82f0-86a1-43e3-8237-846a3aef0c88-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.947344 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.947324    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70b50531-ed83-4bac-bdf1-0992042e01a0-metrics-client-ca\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.947433 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.947340    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.947433 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.947347    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-wtmp\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.947433 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.947421    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-root\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.948055 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.947934    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f7ad82f0-86a1-43e3-8237-846a3aef0c88-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.948055 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.948005    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-sys\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.948245 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.948214    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.950626 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.950602    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.950735 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.950666    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.950909 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.950890    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-tls\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.951476 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.951453    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.965738 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.965708    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc6np\" (UniqueName: \"kubernetes.io/projected/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-api-access-gc6np\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:23.966017 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.965815    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhpx\" (UniqueName: \"kubernetes.io/projected/70b50531-ed83-4bac-bdf1-0992042e01a0-kube-api-access-xqhpx\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:23.984739 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:23.984712    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:24.046274 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:24.046238    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:24.059518 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:36:24.058932    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b50531_ed83_4bac_bdf1_0992042e01a0.slice/crio-b11917bdda887b71d1dbf6a36b422e3b72951d800670d20ac351274ea1216dbd WatchSource:0}: Error finding container b11917bdda887b71d1dbf6a36b422e3b72951d800670d20ac351274ea1216dbd: Status 404 returned error can't find the container with id b11917bdda887b71d1dbf6a36b422e3b72951d800670d20ac351274ea1216dbd
Oct 30 12:36:24.069627 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:24.069599    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:24.297709 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:24.297676    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"]
Oct 30 12:36:24.303641 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:36:24.303605    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7eb0dd3_f31d_4165_a7f8_8ec9594d1cef.slice/crio-10f0c2bc66ec5fbc7ccbaab78b5d9724e568ca1d340056f0243bf684f86f97f0 WatchSource:0}: Error finding container 10f0c2bc66ec5fbc7ccbaab78b5d9724e568ca1d340056f0243bf684f86f97f0: Status 404 returned error can't find the container with id 10f0c2bc66ec5fbc7ccbaab78b5d9724e568ca1d340056f0243bf684f86f97f0
Oct 30 12:36:24.369791 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:24.369761    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"]
Oct 30 12:36:24.767316 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:24.767239    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gvsj" event={"ID":"70b50531-ed83-4bac-bdf1-0992042e01a0","Type":"ContainerStarted","Data":"b11917bdda887b71d1dbf6a36b422e3b72951d800670d20ac351274ea1216dbd"}
Oct 30 12:36:24.769723 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:24.769636    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9" event={"ID":"f7ad82f0-86a1-43e3-8237-846a3aef0c88","Type":"ContainerStarted","Data":"33263084b8c89445ef7119f0efd14350fcdd44baf9a2bff14ba6ded27d2e6685"}
Oct 30 12:36:24.778362 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:24.778180    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w" event={"ID":"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef","Type":"ContainerStarted","Data":"ef6c64dae2f0b30be131183fa223ac49323e1ea99000a93e3ad4920ce10d7508"}
Oct 30 12:36:24.778362 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:24.778300    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w" event={"ID":"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef","Type":"ContainerStarted","Data":"20d08b89ae83f126b9ba1d44da9d2bce9a3f94de90664e83a55ec2076ced6c59"}
Oct 30 12:36:24.778362 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:24.778321    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w" event={"ID":"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef","Type":"ContainerStarted","Data":"10f0c2bc66ec5fbc7ccbaab78b5d9724e568ca1d340056f0243bf684f86f97f0"}
Oct 30 12:36:24.780775 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:24.780738    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zh8sx" event={"ID":"2343d79d-4146-4f1f-90be-ad84a256a4a5","Type":"ContainerStarted","Data":"b565b37491c75f10b8f83db8a03f4eff8c38340a8318d57a0b83793e5e75101e"}
Oct 30 12:36:24.780915 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:24.780797    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zh8sx" event={"ID":"2343d79d-4146-4f1f-90be-ad84a256a4a5","Type":"ContainerStarted","Data":"4b01654eedb55f700691e08095c388d96b840efe6cdd1eaedab29a42b8cbb12d"}
Oct 30 12:36:24.781525 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:24.781490    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:24.821814 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:24.821759    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-zh8sx" podStartSLOduration=2.643191909 podStartE2EDuration="4.821745391s" totalImagesPullingTime="2.178553482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:24.821581001 +0000 UTC m=+44.012469280" watchObservedRunningTime="2026-10-30 12:36:24.821745391 +0000 UTC m=+44.012633661"
Oct 30 12:36:25.050971 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.050662    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:36:25.054362 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.054300    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.069567 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.069522    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 12:36:25.070012 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.069918    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 12:36:25.070135 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.070035    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 12:36:25.070135 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.070077    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 12:36:25.070135 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.070114    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 12:36:25.070135 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.070130    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 12:36:25.070323 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.070150    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 12:36:25.070323 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.070172    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 12:36:25.070323 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.070269    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 12:36:25.070323 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.070278    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dtxmh\""
Oct 30 12:36:25.079876 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.079850    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:36:25.164337 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.164308    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.164518 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.164351    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.164581 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.164556    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-web-config\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.164663 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.164643    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.164731 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.164679    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.164731 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.164711    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.164812 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.164796    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.164863 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.164828    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.164921 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.164898    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-out\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.164972 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.164945    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-volume\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.165024 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.164974    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.165076 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.165056    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqncb\" (UniqueName: \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-kube-api-access-vqncb\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.165126 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.165092    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.266404 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.266333    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-web-config\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.266582 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.266473    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.266582 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.266516    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.266582 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.266548    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.266739 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.266584    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.266739 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.266613    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.266739 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.266649    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-out\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.266739 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.266685    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-volume\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.266739 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.266723    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.266933 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.266772    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqncb\" (UniqueName: \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-kube-api-access-vqncb\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.266933 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.266826    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.266933 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.266875    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.266933 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.266915    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.268539 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.268048    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.269864 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.268779    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.269864 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.269802    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.285802 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.284561    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.285802 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.284575    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.285802 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.284561    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.285802 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.284940    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.285802 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.285334    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-volume\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.285802 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.285735    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-out\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.286202 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.286152    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-web-config\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.287347 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.287291    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqncb\" (UniqueName: \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-kube-api-access-vqncb\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.288125 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.288088    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.293830 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.293793    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.378483 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.378443    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:25.727742 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.727548    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:36:25.739599 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:36:25.739572    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b16518_f7fa_4ddf_ab14_ace75b8d127b.slice/crio-32aa1f56c8be8accf7c779bf4f78e54aa34b11c7b8f4f503d9eaf693fb991370 WatchSource:0}: Error finding container 32aa1f56c8be8accf7c779bf4f78e54aa34b11c7b8f4f503d9eaf693fb991370: Status 404 returned error can't find the container with id 32aa1f56c8be8accf7c779bf4f78e54aa34b11c7b8f4f503d9eaf693fb991370
Oct 30 12:36:25.788072 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.788033    2587 generic.go:356] "Generic (PLEG): container finished" podID="70b50531-ed83-4bac-bdf1-0992042e01a0" containerID="b62051c63b13e1269a481eb91c27900245f92a098f78ec6f01d687d908882914" exitCode=0
Oct 30 12:36:25.788944 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.788118    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gvsj" event={"ID":"70b50531-ed83-4bac-bdf1-0992042e01a0","Type":"ContainerDied","Data":"b62051c63b13e1269a481eb91c27900245f92a098f78ec6f01d687d908882914"}
Oct 30 12:36:25.790046 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.790016    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qmqdw" event={"ID":"ddd38ac5-1413-4345-9452-05936c2b31b8","Type":"ContainerStarted","Data":"5deae7b6251682ba3da5ede879711fb733050be832b8a3687ea21c773e0c6193"}
Oct 30 12:36:25.792830 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.792807    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w" event={"ID":"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef","Type":"ContainerStarted","Data":"4720cecf7debbfe3c21d8c71d23d491d87efb9fb54d92cae0dff572bc98288a6"}
Oct 30 12:36:25.794202 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.794177    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b16518-f7fa-4ddf-ab14-ace75b8d127b","Type":"ContainerStarted","Data":"32aa1f56c8be8accf7c779bf4f78e54aa34b11c7b8f4f503d9eaf693fb991370"}
Oct 30 12:36:25.796679 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.796654    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dg6tk" event={"ID":"e383bb23-4967-47e8-afdf-937f840aea60","Type":"ContainerStarted","Data":"7c3b827c70303123e4f9c5f9feaf77943127787eb6f2b9821abf1d2244c7bd68"}
Oct 30 12:36:25.874885 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.874788    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dg6tk" podStartSLOduration=2.6849231 podStartE2EDuration="5.874772687s" totalImagesPullingTime="3.189849587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:36:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:25.874264235 +0000 UTC m=+45.065152515" watchObservedRunningTime="2026-10-30 12:36:25.874772687 +0000 UTC m=+45.065660965"
Oct 30 12:36:25.910478 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.903192    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-56878775cd-4pz7n"]
Oct 30 12:36:25.915615 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.915588    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:25.916138 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.916079    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w" podStartSLOduration=1.677752522 podStartE2EDuration="2.916061896s" totalImagesPullingTime="1.238309374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:25.914815791 +0000 UTC m=+45.105704072" watchObservedRunningTime="2026-10-30 12:36:25.916061896 +0000 UTC m=+45.106950175"
Oct 30 12:36:25.926596 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.925558    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56878775cd-4pz7n"]
Oct 30 12:36:25.931764 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.931742    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 12:36:25.931913 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.931742    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 12:36:25.931913 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.931821    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 12:36:25.931913 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.931879    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d504fh0uqulkd\""
Oct 30 12:36:25.932068 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.931938    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-w8qvh\""
Oct 30 12:36:25.932068 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.932010    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 12:36:25.932068 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.932026    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 12:36:25.993044 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:25.993002    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qmqdw" podStartSLOduration=2.104525381 podStartE2EDuration="5.992986337s" totalImagesPullingTime="3.888460956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:25.95230532 +0000 UTC m=+45.143193599" watchObservedRunningTime="2026-10-30 12:36:25.992986337 +0000 UTC m=+45.183874613"
Oct 30 12:36:26.074642 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.074606    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.074642 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.074642    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ecfafe4-94c0-451e-b646-54aa9dce7f46-metrics-client-ca\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.074837 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.074676    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecfafe4-94c0-451e-b646-54aa9dce7f46-tmp\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.074837 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.074794    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-grpc-tls\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.074905 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.074833    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.074905 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.074862    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-tls\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.074965 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.074906    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.074965 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.074942    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.075035 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.074981    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8w42\" (UniqueName: \"kubernetes.io/projected/9ecfafe4-94c0-451e-b646-54aa9dce7f46-kube-api-access-z8w42\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.175678 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.175641    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.175678 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.175684    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ecfafe4-94c0-451e-b646-54aa9dce7f46-metrics-client-ca\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.175910 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.175717    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecfafe4-94c0-451e-b646-54aa9dce7f46-tmp\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.175910 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.175747    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-grpc-tls\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.175910 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.175810    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.175910 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.175846    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-tls\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.175910 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.175881    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.176120 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.175922    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.176120 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.175966    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8w42\" (UniqueName: \"kubernetes.io/projected/9ecfafe4-94c0-451e-b646-54aa9dce7f46-kube-api-access-z8w42\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.176652 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.176265    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecfafe4-94c0-451e-b646-54aa9dce7f46-tmp\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.177739 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.176949    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ecfafe4-94c0-451e-b646-54aa9dce7f46-metrics-client-ca\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.184505 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.184440    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.184505 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.184441    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.184795 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.184525    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.184990 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.184970    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-grpc-tls\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.185066 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.185001    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.185975 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.185948    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-tls\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.195067 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.195046    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8w42\" (UniqueName: \"kubernetes.io/projected/9ecfafe4-94c0-451e-b646-54aa9dce7f46-kube-api-access-z8w42\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.241021 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.240987    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:26.557634 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.557608    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56878775cd-4pz7n"]
Oct 30 12:36:26.559897 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:36:26.559866    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ecfafe4_94c0_451e_b646_54aa9dce7f46.slice/crio-a2917b8ca3190dbc570c0e7e783970b40fe6da05ddb92bfc02d3f55bb1a2d67e WatchSource:0}: Error finding container a2917b8ca3190dbc570c0e7e783970b40fe6da05ddb92bfc02d3f55bb1a2d67e: Status 404 returned error can't find the container with id a2917b8ca3190dbc570c0e7e783970b40fe6da05ddb92bfc02d3f55bb1a2d67e
Oct 30 12:36:26.803385 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.803195    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gvsj" event={"ID":"70b50531-ed83-4bac-bdf1-0992042e01a0","Type":"ContainerStarted","Data":"bf516d3283b255ee7db3458930ae2dc0d9edee209fb486cabec52af0746bcb45"}
Oct 30 12:36:26.803805 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.803462    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gvsj" event={"ID":"70b50531-ed83-4bac-bdf1-0992042e01a0","Type":"ContainerStarted","Data":"9abe46ba109708e944178e00ea8e73e75d75ef9b5578b4e468185787f0d56ecc"}
Oct 30 12:36:26.804668 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.804644    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n" event={"ID":"9ecfafe4-94c0-451e-b646-54aa9dce7f46","Type":"ContainerStarted","Data":"a2917b8ca3190dbc570c0e7e783970b40fe6da05ddb92bfc02d3f55bb1a2d67e"}
Oct 30 12:36:26.807559 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.807139    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9" event={"ID":"f7ad82f0-86a1-43e3-8237-846a3aef0c88","Type":"ContainerStarted","Data":"6a48b301207181074be8988cf02b2bfae0a1c8c976dd961696d9a503cfe8714a"}
Oct 30 12:36:26.807559 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.807191    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9" event={"ID":"f7ad82f0-86a1-43e3-8237-846a3aef0c88","Type":"ContainerStarted","Data":"673e8acf9eb36d449439d3c47fe251fd620af8fdfb7fde8071f2d2a72d70cff5"}
Oct 30 12:36:26.807747 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.807721    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9" event={"ID":"f7ad82f0-86a1-43e3-8237-846a3aef0c88","Type":"ContainerStarted","Data":"97e28a370d2e6afe63ab713351a0374262d5980bb67f091780a2cb92c7ed0a47"}
Oct 30 12:36:26.843879 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.843831    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9gvsj" podStartSLOduration=2.643051044 podStartE2EDuration="3.84381843s" totalImagesPullingTime="1.101410755s" totalInitContainerRuntime="99.356631ms" isStatefulPod=true podCreationTimestamp="2026-10-30 12:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:26.842936065 +0000 UTC m=+46.033824345" watchObservedRunningTime="2026-10-30 12:36:26.84381843 +0000 UTC m=+46.034706710"
Oct 30 12:36:26.885617 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:26.885570    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9" podStartSLOduration=2.416876427 podStartE2EDuration="3.885556899s" totalImagesPullingTime="1.468680472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:26.885294746 +0000 UTC m=+46.076183023" watchObservedRunningTime="2026-10-30 12:36:26.885556899 +0000 UTC m=+46.076445177"
Oct 30 12:36:27.812271 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:27.812234    2587 generic.go:356] "Generic (PLEG): container finished" podID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerID="efa532c5fe9cd30ec8a7deefcad58c09f85970b879b921fea96683d75c24e165" exitCode=0
Oct 30 12:36:27.812804 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:27.812319    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b16518-f7fa-4ddf-ab14-ace75b8d127b","Type":"ContainerDied","Data":"efa532c5fe9cd30ec8a7deefcad58c09f85970b879b921fea96683d75c24e165"}
Oct 30 12:36:28.114943 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.114844    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6f9858f786-6dftr"]
Oct 30 12:36:28.117771 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.117748    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.135808 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.135782    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 12:36:28.136284 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.135836    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 12:36:28.137140 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.137117    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f9858f786-6dftr"]
Oct 30 12:36:28.144093 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.144064    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 12:36:28.144239 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.144159    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 12:36:28.146801 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.146781    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6dg1ritap5cqn\""
Oct 30 12:36:28.146920 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.146861    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zf26c\""
Oct 30 12:36:28.194789 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.194745    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7709b8a5-ca78-45d6-9c8c-136a13f50c72-audit-log\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.194992 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.194857    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l6nw\" (UniqueName: \"kubernetes.io/projected/7709b8a5-ca78-45d6-9c8c-136a13f50c72-kube-api-access-8l6nw\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.194992 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.194893    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-client-ca-bundle\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.195085 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.195070    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-secret-metrics-server-client-certs\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.195116 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.195100    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7709b8a5-ca78-45d6-9c8c-136a13f50c72-metrics-server-audit-profiles\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.195160 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.195141    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-secret-metrics-server-tls\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.195197 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.195177    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7709b8a5-ca78-45d6-9c8c-136a13f50c72-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.296211 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.296166    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-secret-metrics-server-tls\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.296211 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.296207    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7709b8a5-ca78-45d6-9c8c-136a13f50c72-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.296483 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.296242    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7709b8a5-ca78-45d6-9c8c-136a13f50c72-audit-log\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.296483 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.296274    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l6nw\" (UniqueName: \"kubernetes.io/projected/7709b8a5-ca78-45d6-9c8c-136a13f50c72-kube-api-access-8l6nw\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.296483 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.296301    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-client-ca-bundle\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.296483 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.296430    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-secret-metrics-server-client-certs\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.296483 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.296465    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7709b8a5-ca78-45d6-9c8c-136a13f50c72-metrics-server-audit-profiles\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.296983 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.296933    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7709b8a5-ca78-45d6-9c8c-136a13f50c72-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.297240 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.297214    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7709b8a5-ca78-45d6-9c8c-136a13f50c72-audit-log\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.297540 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.297476    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7709b8a5-ca78-45d6-9c8c-136a13f50c72-metrics-server-audit-profiles\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.299952 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.299921    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-secret-metrics-server-tls\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.303221 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.303197    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-secret-metrics-server-client-certs\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.306892 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.306867    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-client-ca-bundle\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.316271 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.316245    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l6nw\" (UniqueName: \"kubernetes.io/projected/7709b8a5-ca78-45d6-9c8c-136a13f50c72-kube-api-access-8l6nw\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.430725 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.430648    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:28.460882 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.459465    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"]
Oct 30 12:36:28.463127 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.463096    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"
Oct 30 12:36:28.480156 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.480099    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"]
Oct 30 12:36:28.483147 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.482639    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-q874n\""
Oct 30 12:36:28.483147 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.482724    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 12:36:28.498055 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.498019    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtpt8\" (UniqueName: \"kubernetes.io/projected/30a15bca-3504-49fe-9ecf-c9860108905e-kube-api-access-mtpt8\") pod \"monitoring-plugin-77f4f76594-wqwgs\" (UID: \"30a15bca-3504-49fe-9ecf-c9860108905e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"
Oct 30 12:36:28.498336 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.498302    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30a15bca-3504-49fe-9ecf-c9860108905e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wqwgs\" (UID: \"30a15bca-3504-49fe-9ecf-c9860108905e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"
Oct 30 12:36:28.599780 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.599742    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpt8\" (UniqueName: \"kubernetes.io/projected/30a15bca-3504-49fe-9ecf-c9860108905e-kube-api-access-mtpt8\") pod \"monitoring-plugin-77f4f76594-wqwgs\" (UID: \"30a15bca-3504-49fe-9ecf-c9860108905e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"
Oct 30 12:36:28.599891 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.599801    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30a15bca-3504-49fe-9ecf-c9860108905e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wqwgs\" (UID: \"30a15bca-3504-49fe-9ecf-c9860108905e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"
Oct 30 12:36:28.602290 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.602265    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30a15bca-3504-49fe-9ecf-c9860108905e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wqwgs\" (UID: \"30a15bca-3504-49fe-9ecf-c9860108905e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"
Oct 30 12:36:28.618157 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.618129    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpt8\" (UniqueName: \"kubernetes.io/projected/30a15bca-3504-49fe-9ecf-c9860108905e-kube-api-access-mtpt8\") pod \"monitoring-plugin-77f4f76594-wqwgs\" (UID: \"30a15bca-3504-49fe-9ecf-c9860108905e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"
Oct 30 12:36:28.751292 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.751256    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f9858f786-6dftr"]
Oct 30 12:36:28.778531 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.778504    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"
Oct 30 12:36:28.817418 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.817358    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f9858f786-6dftr" event={"ID":"7709b8a5-ca78-45d6-9c8c-136a13f50c72","Type":"ContainerStarted","Data":"82b27b3ae3931467c782820e573951472ce7db04f7d88f93702fb9a69fb06a00"}
Oct 30 12:36:28.823018 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.822986    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n" event={"ID":"9ecfafe4-94c0-451e-b646-54aa9dce7f46","Type":"ContainerStarted","Data":"42c541b365e12c0ef64dd47eb7fc3ebf782dbcfcf5c8c9f3573097da53b23cd2"}
Oct 30 12:36:28.823160 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.823068    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n" event={"ID":"9ecfafe4-94c0-451e-b646-54aa9dce7f46","Type":"ContainerStarted","Data":"05c438bacb14edbd1c239b556a4ef2b0ff4aa4fe425224bbecbadc1c2a291aed"}
Oct 30 12:36:28.823160 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:28.823110    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n" event={"ID":"9ecfafe4-94c0-451e-b646-54aa9dce7f46","Type":"ContainerStarted","Data":"f1b33cd774b9ca65ad0d57f91a1f2dfadec3013a5d8246c0ff6973e7bbe9bb71"}
Oct 30 12:36:29.072308 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:29.072266    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"]
Oct 30 12:36:29.077054 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:36:29.077012    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a15bca_3504_49fe_9ecf_c9860108905e.slice/crio-e61575334e78a38be03a8b23bd58bc7c4937da4b4554f8bb54869a7768284963 WatchSource:0}: Error finding container e61575334e78a38be03a8b23bd58bc7c4937da4b4554f8bb54869a7768284963: Status 404 returned error can't find the container with id e61575334e78a38be03a8b23bd58bc7c4937da4b4554f8bb54869a7768284963
Oct 30 12:36:29.830885 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:29.830847    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n" event={"ID":"9ecfafe4-94c0-451e-b646-54aa9dce7f46","Type":"ContainerStarted","Data":"2dc7f84060256f94dd8c88ed51f13c7f689de708af26734da4b52e77e64bc040"}
Oct 30 12:36:29.831365 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:29.830925    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n" event={"ID":"9ecfafe4-94c0-451e-b646-54aa9dce7f46","Type":"ContainerStarted","Data":"268994b71a8cb06ede0896653079b24ab393c90b796fc8fa40981da7460e0ea8"}
Oct 30 12:36:29.831365 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:29.830964    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n" event={"ID":"9ecfafe4-94c0-451e-b646-54aa9dce7f46","Type":"ContainerStarted","Data":"b2c0e4f41de6ae1500d590fa9581852483771c311d8ea4481990afd4e4a5b6ff"}
Oct 30 12:36:29.831531 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:29.831496    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:29.833102 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:29.833073    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs" event={"ID":"30a15bca-3504-49fe-9ecf-c9860108905e","Type":"ContainerStarted","Data":"e61575334e78a38be03a8b23bd58bc7c4937da4b4554f8bb54869a7768284963"}
Oct 30 12:36:29.837829 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:29.837804    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b16518-f7fa-4ddf-ab14-ace75b8d127b","Type":"ContainerStarted","Data":"ba0cde52853918504e8e2b02c015fdf789a1964c1f93caa408dd1eeb878803ad"}
Oct 30 12:36:29.837962 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:29.837872    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b16518-f7fa-4ddf-ab14-ace75b8d127b","Type":"ContainerStarted","Data":"01cb6c1f41abb8ee8d23626a841926d92a98a3abfa4fc270a32683230d6eda2f"}
Oct 30 12:36:29.837962 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:29.837904    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b16518-f7fa-4ddf-ab14-ace75b8d127b","Type":"ContainerStarted","Data":"7d6311147738646a2b8099040a3efa4c265f0939d1d144f571e3ad1df8c54953"}
Oct 30 12:36:29.837962 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:29.837927    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b16518-f7fa-4ddf-ab14-ace75b8d127b","Type":"ContainerStarted","Data":"a22b4e94bbcd95169aed0f2e64ee27c20b883de077763946b807e9840fca4732"}
Oct 30 12:36:29.837962 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:29.837956    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b16518-f7fa-4ddf-ab14-ace75b8d127b","Type":"ContainerStarted","Data":"c101f9bdf653ede0666117b92796b7a8f3855231edb0ac2305d73f3b5e528e36"}
Oct 30 12:36:29.838091 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:29.837981    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b16518-f7fa-4ddf-ab14-ace75b8d127b","Type":"ContainerStarted","Data":"c00678908b8509631d5735197a35b3c60d7c5b2e9b6e1a1da46fc5043089fcc0"}
Oct 30 12:36:29.878014 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:29.877897    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n" podStartSLOduration=2.185320114 podStartE2EDuration="4.877877377s" totalImagesPullingTime="2.692557263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:29.876126194 +0000 UTC m=+49.067014475" watchObservedRunningTime="2026-10-30 12:36:29.877877377 +0000 UTC m=+49.068765657"
Oct 30 12:36:29.925477 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:29.925415    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.389057983 podStartE2EDuration="4.925378957s" totalImagesPullingTime="2.519036971s" totalInitContainerRuntime="17.284003ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:29.923769095 +0000 UTC m=+49.114657399" watchObservedRunningTime="2026-10-30 12:36:29.925378957 +0000 UTC m=+49.116267239"
Oct 30 12:36:30.484689 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.484650    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:36:30.488434 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.488404    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.518456 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.518419    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 12:36:30.518601 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.518527    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 12:36:30.518601 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.518552    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 12:36:30.518786 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.518767    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 12:36:30.518855 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.518810    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 12:36:30.518855 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.518822    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 12:36:30.521272 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.521250    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.521368 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.521282    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.521368 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.521307    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.521480 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.521375    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.521480 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.521456    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.521562 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.521494    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.521629 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.521610    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.521662 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.521637    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.521705 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.521679    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.521748 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.521733    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.521861 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.521768    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d57bx\" (UniqueName: \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-kube-api-access-d57bx\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.521861 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.521836    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.521958 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.521875    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.521958 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.521932    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.522028 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.521968    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.522028 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.522020    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.522118 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.522038    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.522118 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.522067    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.522118 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.522107    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.522241 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.522151    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.525889 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.525869    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1rld844cc7msd\""
Oct 30 12:36:30.526111 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.526096    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 12:36:30.526200 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.526173    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 12:36:30.526200 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.526186    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wbnrm\""
Oct 30 12:36:30.526297 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.526205    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 12:36:30.530183 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.530168    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 12:36:30.530241 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.530172    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 12:36:30.530621 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.530609    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 12:36:30.543022 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.542995    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 12:36:30.549466 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.549445    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 12:36:30.552869 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.552847    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:36:30.623341 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.623266    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.623341 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.623306    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.623341 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.623325    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.623341 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.623343    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624339 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.623842    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624339 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.623883    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624339 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.623933    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624339 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.623958    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624339 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.623985    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624339 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.623989    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624339 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.624015    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624339 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.624203    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624339 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.624236    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624339 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.624302    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624339 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.624325    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624909 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.624377    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624909 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.624419    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624909 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.624455    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d57bx\" (UniqueName: \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-kube-api-access-d57bx\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624909 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.624483    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624909 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.624518    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624909 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.624557    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.624909 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.624865    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.626001 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.625973    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.626107 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.626061    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.628149 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.626573    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.628149 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.627294    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.628149 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.627711    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.628149 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.627712    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.628149 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.628096    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.628774 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.628751    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.629009 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.628986    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.629278 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.629250    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.629594 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.629550    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.629947 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.629921    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.630034 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.630019    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.630902 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.630878    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.630995 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.630944    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.631245 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.631224    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.631575 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.631547    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.643820 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.643791    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d57bx\" (UniqueName: \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-kube-api-access-d57bx\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.809706 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.809674    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:30.842891 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.842851    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f9858f786-6dftr" event={"ID":"7709b8a5-ca78-45d6-9c8c-136a13f50c72","Type":"ContainerStarted","Data":"249ead2cc59bb4c542246eaee8f218232f392392eeac35d23bdb43d21d129eeb"}
Oct 30 12:36:30.844465 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.844432    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs" event={"ID":"30a15bca-3504-49fe-9ecf-c9860108905e","Type":"ContainerStarted","Data":"e38be6d473442ba84e51a9bcafaaf45b7d2ebd3ba291dbd6346d1726dd32b004"}
Oct 30 12:36:30.845115 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.845067    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"
Oct 30 12:36:30.850571 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.850550    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"
Oct 30 12:36:30.882554 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.882462    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6f9858f786-6dftr" podStartSLOduration=1.268489766 podStartE2EDuration="2.882441518s" totalImagesPullingTime="1.613951752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:30.881773834 +0000 UTC m=+50.072662112" watchObservedRunningTime="2026-10-30 12:36:30.882441518 +0000 UTC m=+50.073329798"
Oct 30 12:36:30.918326 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:30.918245    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs" podStartSLOduration=1.4790410139999999 podStartE2EDuration="2.918231424s" totalImagesPullingTime="1.43919041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:30.917678065 +0000 UTC m=+50.108566343" watchObservedRunningTime="2026-10-30 12:36:30.918231424 +0000 UTC m=+50.109119703"
Oct 30 12:36:31.124529 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:31.124507    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:36:31.125814 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:36:31.125786    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1b5758_6ef0_452a_8ca7_c9457e30e60a.slice/crio-b0ddff1458fcd7794a111b51a4622cf59e3f3a08b52c2cf78ea14f6adbae9e64 WatchSource:0}: Error finding container b0ddff1458fcd7794a111b51a4622cf59e3f3a08b52c2cf78ea14f6adbae9e64: Status 404 returned error can't find the container with id b0ddff1458fcd7794a111b51a4622cf59e3f3a08b52c2cf78ea14f6adbae9e64
Oct 30 12:36:31.850121 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:31.850089    2587 generic.go:356] "Generic (PLEG): container finished" podID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerID="d12dfc6d683790f1f80da3c4b6ef5105a72f9cea3026d150db56d8645f6d3206" exitCode=0
Oct 30 12:36:31.850510 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:31.850178    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c1b5758-6ef0-452a-8ca7-c9457e30e60a","Type":"ContainerDied","Data":"d12dfc6d683790f1f80da3c4b6ef5105a72f9cea3026d150db56d8645f6d3206"}
Oct 30 12:36:31.850510 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:31.850246    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c1b5758-6ef0-452a-8ca7-c9457e30e60a","Type":"ContainerStarted","Data":"b0ddff1458fcd7794a111b51a4622cf59e3f3a08b52c2cf78ea14f6adbae9e64"}
Oct 30 12:36:34.798954 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:34.798927    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:34.872552 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:34.865459    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c1b5758-6ef0-452a-8ca7-c9457e30e60a","Type":"ContainerStarted","Data":"3aae098487a7adfbbbfc49d48f1ee310cf1c1a8b1d35a6f37c35859f72cf7370"}
Oct 30 12:36:34.872552 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:34.865557    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c1b5758-6ef0-452a-8ca7-c9457e30e60a","Type":"ContainerStarted","Data":"8ae65fb3e8b6f0bcb64a377fc741e4525198029cd4af0b6558b671f4245c41c1"}
Oct 30 12:36:35.851875 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:35.851846    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:35.872833 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:35.872803    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c1b5758-6ef0-452a-8ca7-c9457e30e60a","Type":"ContainerStarted","Data":"05fb4d7ee38cd3b282c0a2d30bf057d8b7682ee56635366dbb40d079bb99cae1"}
Oct 30 12:36:35.873013 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:35.872877    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c1b5758-6ef0-452a-8ca7-c9457e30e60a","Type":"ContainerStarted","Data":"45b3f0de2d73c85f7e7c3aa864396d3c74d1347c5c17141e8ca51bcf772dbec9"}
Oct 30 12:36:35.873013 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:35.872920    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c1b5758-6ef0-452a-8ca7-c9457e30e60a","Type":"ContainerStarted","Data":"0142684f956bba4a10402aed864af544d0a20a30e765d31248d59ed93f7fc59e"}
Oct 30 12:36:35.873013 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:35.872959    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c1b5758-6ef0-452a-8ca7-c9457e30e60a","Type":"ContainerStarted","Data":"244e96c496fa0b9dc5cb346fac30ac43d313a45a8ad502521fb1bfe113695c24"}
Oct 30 12:36:35.945629 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:35.945578    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.10504719 podStartE2EDuration="5.945564653s" totalImagesPullingTime="2.823311556s" totalInitContainerRuntime="17.205907ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:35.939776148 +0000 UTC m=+55.130664460" watchObservedRunningTime="2026-10-30 12:36:35.945564653 +0000 UTC m=+55.136452932"
Oct 30 12:36:39.326603 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:39.326573    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:40.810332 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:40.810295    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:46.069299 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.069258    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:46.069299 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.069300    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:46.082327 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.082299    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:36:46.092178 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.092147    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:46.093722 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.093702    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:36:46.101910 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.101888    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:46.118093 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.118069    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bg2k7\""
Oct 30 12:36:46.125133 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.125116    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:46.153761 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.153740    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-54vc4\""
Oct 30 12:36:46.160894 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.160867    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:46.170831 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.170796    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhm4h\" (UniqueName: \"kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h\") pod \"network-check-target-chqf2\" (UID: \"cd722809-68b9-41b8-8a5e-814903a74299\") " pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:46.183825 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.183805    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:36:46.204819 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.204790    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:36:46.214445 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.214416    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhm4h\" (UniqueName: \"kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h\") pod \"network-check-target-chqf2\" (UID: \"cd722809-68b9-41b8-8a5e-814903a74299\") " pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:46.401161 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.401084    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-t6s7j\""
Oct 30 12:36:46.409126 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.409095    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:46.417581 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.417553    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kd88h"]
Oct 30 12:36:46.420213 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:36:46.420181    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b894531_bbb4_4e28_b4d4_ac51d5ed4ca4.slice/crio-56079d226920cb26fe205a7aacbc2f3caa3e27655d3ca5172c0791fe7e354637 WatchSource:0}: Error finding container 56079d226920cb26fe205a7aacbc2f3caa3e27655d3ca5172c0791fe7e354637: Status 404 returned error can't find the container with id 56079d226920cb26fe205a7aacbc2f3caa3e27655d3ca5172c0791fe7e354637
Oct 30 12:36:46.469324 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.469293    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v4ttl"]
Oct 30 12:36:46.472527 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:36:46.472483    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7d1b97_ad0b_4e7f_b2e2_b82bbd76fe8b.slice/crio-b74e6a2dbfaaf708abc9916c24f331f1b9a7b0a6f7e675af08053d564ddbacd3 WatchSource:0}: Error finding container b74e6a2dbfaaf708abc9916c24f331f1b9a7b0a6f7e675af08053d564ddbacd3: Status 404 returned error can't find the container with id b74e6a2dbfaaf708abc9916c24f331f1b9a7b0a6f7e675af08053d564ddbacd3
Oct 30 12:36:46.702911 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.702884    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-chqf2"]
Oct 30 12:36:46.703838 ip-10-0-137-228 kubenswrapper[2587]: W1030 12:36:46.703802    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd722809_68b9_41b8_8a5e_814903a74299.slice/crio-9021c0a7ce94ad61b563080fc923ef0bb59486bf0102408a9c0969d70222e5cd WatchSource:0}: Error finding container 9021c0a7ce94ad61b563080fc923ef0bb59486bf0102408a9c0969d70222e5cd: Status 404 returned error can't find the container with id 9021c0a7ce94ad61b563080fc923ef0bb59486bf0102408a9c0969d70222e5cd
Oct 30 12:36:46.916463 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.916423    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-chqf2" event={"ID":"cd722809-68b9-41b8-8a5e-814903a74299","Type":"ContainerStarted","Data":"9021c0a7ce94ad61b563080fc923ef0bb59486bf0102408a9c0969d70222e5cd"}
Oct 30 12:36:46.918138 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.918100    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v4ttl" event={"ID":"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b","Type":"ContainerStarted","Data":"b74e6a2dbfaaf708abc9916c24f331f1b9a7b0a6f7e675af08053d564ddbacd3"}
Oct 30 12:36:46.919616 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:46.919582    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kd88h" event={"ID":"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4","Type":"ContainerStarted","Data":"56079d226920cb26fe205a7aacbc2f3caa3e27655d3ca5172c0791fe7e354637"}
Oct 30 12:36:47.929295 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:47.929259    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v4ttl" event={"ID":"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b","Type":"ContainerStarted","Data":"f15b32964c81adf35fc6939e4ba894221246870f56727a2b17ff0094dbf9843b"}
Oct 30 12:36:48.431092 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:48.431053    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:48.431273 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:48.431130    2587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:48.936387 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:48.936353    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v4ttl" event={"ID":"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b","Type":"ContainerStarted","Data":"a97635ca89963f92fd6a611a3613b7eb3600ecc85e85e13f3583717a9a503985"}
Oct 30 12:36:48.976435 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:48.976353    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v4ttl" podStartSLOduration=66.742598381 podStartE2EDuration="1m7.976331594s" totalImagesPullingTime="1.233733213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:48.975585129 +0000 UTC m=+68.166473429" watchObservedRunningTime="2026-10-30 12:36:48.976331594 +0000 UTC m=+68.167219875"
Oct 30 12:36:50.950320 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:50.950279    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-chqf2" event={"ID":"cd722809-68b9-41b8-8a5e-814903a74299","Type":"ContainerStarted","Data":"4d85ab13401902e040771f101262037626189e7a30ac713af7125dcfd548cabf"}
Oct 30 12:36:50.950798 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:50.950355    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:50.988302 ip-10-0-137-228 kubenswrapper[2587]: I1030 12:36:50.988236    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-chqf2" podStartSLOduration=66.710450988 podStartE2EDuration="1m9.988213657s" totalImagesPullingTime="3.277762669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:50.987175425 +0000 UTC m=+70.178063705" watchObservedRunningTime="2026-10-30 12:36:50.988213657 +0000 UTC m=+70.179101939"
Oct 30 12:36:51.085107 ip-10-0-137-228 kubenswrapper[2587]: E1030 12:36:51.085067    2587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:36:51.411456 ip-10-0-137-228 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 12:36:51.428929 ip-10-0-137-228 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 12:36:51.429110 ip-10-0-137-228 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 12:36:51.429323 ip-10-0-137-228 systemd[1]: kubelet.service: Consumed 4.693s CPU time.
Oct 30 12:36:51.440881 ip-10-0-137-228 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 12:36:51.508524 ip-10-0-137-228 kubenswrapper[5837]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:36:51.508524 ip-10-0-137-228 kubenswrapper[5837]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 12:36:51.508524 ip-10-0-137-228 kubenswrapper[5837]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:36:51.508524 ip-10-0-137-228 kubenswrapper[5837]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:36:51.510859 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510758    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:36:51.510859 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510856    5837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:36:51.510859 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510861    5837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:36:51.510859 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510864    5837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:36:51.510859 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510867    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510870    5837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510872    5837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510875    5837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510885    5837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510889    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510892    5837 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510895    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510898    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510900    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510903    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510905    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510907    5837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510910    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510912    5837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510915    5837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510919    5837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510924    5837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510927    5837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:36:51.511064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510929    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510931    5837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510935    5837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510938    5837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510940    5837 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510943    5837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510945    5837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510948    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510951    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510953    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510956    5837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510959    5837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510961    5837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510964    5837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510966    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510968    5837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510977    5837 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510980    5837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510982    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:36:51.511576 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510985    5837 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510988    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510991    5837 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510993    5837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510996    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.510998    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511000    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511003    5837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511005    5837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511008    5837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511010    5837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511013    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511015    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511018    5837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511020    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511024    5837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511026    5837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511029    5837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511031    5837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511034    5837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:36:51.512107 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511036    5837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511040    5837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511042    5837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511045    5837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511047    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511050    5837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511052    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511054    5837 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511057    5837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511059    5837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511063    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511066    5837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511068    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511071    5837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511074    5837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511077    5837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511079    5837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511082    5837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511084    5837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:36:51.512637 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511087    5837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511091    5837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511094    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511098    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511101    5837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511103    5837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511106    5837 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511108    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511111    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511114    5837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511116    5837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511119    5837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511121    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511124    5837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511126    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511129    5837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511131    5837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511133    5837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511135    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:36:51.513108 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511138    5837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511140    5837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511142    5837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511145    5837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511147    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511150    5837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511957    5837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511964    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511968    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511971    5837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511974    5837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511976    5837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511979    5837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511982    5837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511984    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511988    5837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511990    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511993    5837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511995    5837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.511998    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:36:51.513618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512000    5837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512003    5837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512006    5837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512008    5837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512011    5837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512014    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512016    5837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512019    5837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512021    5837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512024    5837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512026    5837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512029    5837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512031    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512034    5837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512037    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512039    5837 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512042    5837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512045    5837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512048    5837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512051    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:36:51.514160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512054    5837 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512057    5837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512059    5837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512061    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512064    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512066    5837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512069    5837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512071    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512074    5837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512076    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512079    5837 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512081    5837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512083    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512086    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512088    5837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512091    5837 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512093    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512096    5837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512098    5837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512101    5837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:36:51.514679 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512103    5837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512106    5837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512109    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512120    5837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512124    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512127    5837 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512134    5837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512137    5837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512139    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512142    5837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512145    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512148    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512150    5837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512153    5837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512155    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512158    5837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512160    5837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512163    5837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512165    5837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:36:51.515176 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512168    5837 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512171    5837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512173    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512177    5837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512179    5837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512182    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512184    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512186    5837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512189    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512191    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512194    5837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512196    5837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512199    5837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512201    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512204    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512206    5837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512208    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512211    5837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512213    5837 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:36:51.515686 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512217    5837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512220    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512222    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512225    5837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512227    5837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512229    5837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512233    5837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512235    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512238    5837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512240    5837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512242    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512245    5837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512249    5837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512253    5837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512812    5837 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512818    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512821    5837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512824    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512827    5837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512829    5837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:36:51.516151 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512832    5837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512835    5837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512837    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512840    5837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512843    5837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512845    5837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512848    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512850    5837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512853    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512855    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512858    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512860    5837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512863    5837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512866    5837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512868    5837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512871    5837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512873    5837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512876    5837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512879    5837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512882    5837 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:36:51.516666 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512884    5837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512886    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512889    5837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512892    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512894    5837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512897    5837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512900    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512903    5837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512906    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512908    5837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512911    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512913    5837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512916    5837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512918    5837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512920    5837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512923    5837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512926    5837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512928    5837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512930    5837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:36:51.517185 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512933    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512937    5837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512940    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512943    5837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512946    5837 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512948    5837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512951    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512953    5837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512956    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512958    5837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512961    5837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512963    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512966    5837 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512969    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512972    5837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512974    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512977    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512979    5837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512982    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:36:51.517723 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512985    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512989    5837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512993    5837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512996    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.512998    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513001    5837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513003    5837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513006    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513009    5837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513011    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513013    5837 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513016    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513018    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513021    5837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513023    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513025    5837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513028    5837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513030    5837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513033    5837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:36:51.518225 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513035    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513037    5837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513041    5837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513043    5837 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513045    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513048    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513050    5837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513053    5837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513056    5837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513058    5837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513061    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513063    5837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513066    5837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513068    5837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513070    5837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513075    5837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513078    5837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513080    5837 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513082    5837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:36:51.518734 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513085    5837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513087    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513091    5837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513093    5837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513186    5837 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513194    5837 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513200    5837 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513205    5837 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513209    5837 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513212    5837 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513216    5837 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513222    5837 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513225    5837 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513228    5837 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513232    5837 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513235    5837 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513238    5837 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513241    5837 flags.go:64] FLAG: --cgroup-root=""
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513246    5837 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513249    5837 flags.go:64] FLAG: --client-ca-file=""
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513252    5837 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513256    5837 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513260    5837 flags.go:64] FLAG: --cluster-domain=""
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513263    5837 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 12:36:51.519213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513266    5837 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513269    5837 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513272    5837 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513277    5837 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513281    5837 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513284    5837 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513288    5837 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513290    5837 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513294    5837 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513297    5837 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513300    5837 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513303    5837 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513307    5837 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513311    5837 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513314    5837 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513316    5837 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513320    5837 flags.go:64] FLAG: --enable-server="true"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513322    5837 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513327    5837 flags.go:64] FLAG: --event-burst="100"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513330    5837 flags.go:64] FLAG: --event-qps="50"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513333    5837 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513336    5837 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513339    5837 flags.go:64] FLAG: --eviction-hard=""
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513343    5837 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 12:36:51.519962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513346    5837 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513349    5837 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513353    5837 flags.go:64] FLAG: --eviction-soft=""
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513356    5837 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513359    5837 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513362    5837 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513366    5837 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513369    5837 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513372    5837 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513374    5837 flags.go:64] FLAG: --feature-gates=""
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513379    5837 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513382    5837 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513407    5837 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513411    5837 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513414    5837 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513417    5837 flags.go:64] FLAG: --help="false"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513420    5837 flags.go:64] FLAG: --hostname-override="ip-10-0-137-228.ec2.internal"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513423    5837 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513427    5837 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513430    5837 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513433    5837 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513437    5837 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513439    5837 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 12:36:51.520701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513442    5837 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513445    5837 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513448    5837 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513451    5837 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513454    5837 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513457    5837 flags.go:64] FLAG: --kube-reserved=""
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513460    5837 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513462    5837 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513466    5837 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513468    5837 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513471    5837 flags.go:64] FLAG: --lock-file=""
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513475    5837 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513478    5837 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513481    5837 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513487    5837 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513490    5837 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513494    5837 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513497    5837 flags.go:64] FLAG: --logging-format="text"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513500    5837 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513503    5837 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513506    5837 flags.go:64] FLAG: --manifest-url=""
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513510    5837 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513515    5837 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513518    5837 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513522    5837 flags.go:64] FLAG: --max-pods="110"
Oct 30 12:36:51.521330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513526    5837 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513529    5837 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513531    5837 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513535    5837 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513538    5837 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513541    5837 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513544    5837 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513551    5837 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513555    5837 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513558    5837 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513561    5837 flags.go:64] FLAG: --pod-cidr=""
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513564    5837 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513567    5837 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513570    5837 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513573    5837 flags.go:64] FLAG: --port="10250"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513576    5837 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513579    5837 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0df263eda56ea87ab"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513582    5837 flags.go:64] FLAG: --qos-reserved=""
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513586    5837 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513589    5837 flags.go:64] FLAG: --register-node="true"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513592    5837 flags.go:64] FLAG: --register-with-taints=""
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513596    5837 flags.go:64] FLAG: --registry-burst="10"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513599    5837 flags.go:64] FLAG: --registry-qps="5"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513602    5837 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513605    5837 flags.go:64] FLAG: --reserved-memory=""
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513609    5837 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 12:36:51.522062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513612    5837 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513615    5837 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513619    5837 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513622    5837 flags.go:64] FLAG: --runonce="false"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513625    5837 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513628    5837 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513631    5837 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513634    5837 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513637    5837 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513640    5837 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513644    5837 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513647    5837 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513659    5837 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513662    5837 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513665    5837 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513668    5837 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513671    5837 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513674    5837 flags.go:64] FLAG: --system-cgroups=""
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513677    5837 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513683    5837 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513686    5837 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513688    5837 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513695    5837 flags.go:64] FLAG: --tls-min-version=""
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513698    5837 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513703    5837 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 12:36:51.522719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513706    5837 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513709    5837 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513712    5837 flags.go:64] FLAG: --v="2"
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513719    5837 flags.go:64] FLAG: --version="false"
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513723    5837 flags.go:64] FLAG: --vmodule=""
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513727    5837 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.513735    5837 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513881    5837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513886    5837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513891    5837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513895    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513899    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513902    5837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513904    5837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513907    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513910    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513912    5837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513915    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513917    5837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513920    5837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:36:51.523373 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513922    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513925    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513927    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513930    5837 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513932    5837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513935    5837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513937    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513940    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513942    5837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513945    5837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513947    5837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513952    5837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513955    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513957    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513960    5837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513962    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513965    5837 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513968    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513970    5837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:36:51.523907 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513973    5837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513976    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513979    5837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513982    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513984    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513987    5837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513989    5837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513992    5837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513994    5837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.513997    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514000    5837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514002    5837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514004    5837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514007    5837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514009    5837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514012    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514015    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514017    5837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514020    5837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:36:51.524362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514022    5837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514025    5837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514027    5837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514030    5837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514032    5837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514037    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514039    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514042    5837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514044    5837 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514047    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514049    5837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514052    5837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514054    5837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514057    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514060    5837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514062    5837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514065    5837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514067    5837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514069    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:36:51.524918 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514072    5837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514074    5837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514077    5837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514079    5837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514081    5837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514084    5837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514086    5837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514089    5837 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514091    5837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514094    5837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514096    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514098    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514102    5837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514106    5837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514109    5837 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514111    5837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514114    5837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514117    5837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514120    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514123    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:36:51.525386 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514125    5837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514128    5837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514130    5837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514133    5837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514135    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514138    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514141    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514143    5837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514146    5837 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514149    5837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514151    5837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514154    5837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514157    5837 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514162    5837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514166    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514170    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:36:51.525977 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.514197    5837 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514232    5837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514237    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514240    5837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514242    5837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514245    5837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514248    5837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514250    5837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514253    5837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514255    5837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514259    5837 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514262    5837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514264    5837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514267    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514269    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514274    5837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514278    5837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514281    5837 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514283    5837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:36:51.526438 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514286    5837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514288    5837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514292    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514294    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514297    5837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514299    5837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514305    5837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514308    5837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514311    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514314    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514316    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514319    5837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514321    5837 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514324    5837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514326    5837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514329    5837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514331    5837 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514334    5837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514337    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514339    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:36:51.527129 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514341    5837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514344    5837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514348    5837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514353    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514356    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514359    5837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514361    5837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514364    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514368    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514371    5837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514375    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514377    5837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514380    5837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514382    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514385    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514387    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514406    5837 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514409    5837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514411    5837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:36:51.527717 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514414    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514417    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514420    5837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514423    5837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514425    5837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514428    5837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514430    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514433    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514435    5837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514438    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514440    5837 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514443    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514445    5837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514448    5837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514450    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514453    5837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514458    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514460    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514463    5837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:36:51.528186 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514465    5837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514468    5837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514472    5837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514474    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514477    5837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514479    5837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514482    5837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514484    5837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514487    5837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514490    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514493    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514496    5837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514499    5837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514501    5837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514510    5837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514512    5837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514515    5837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514517    5837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514520    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:36:51.528760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514522    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514525    5837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514527    5837 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514530    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514532    5837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514535    5837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514537    5837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514540    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514542    5837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514545    5837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.514549    5837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.519151    5837 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.519169    5837 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.519188    5837 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.519193    5837 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519268    5837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519273    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519276    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519279    5837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519283    5837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:36:51.529380 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519286    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519288    5837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519291    5837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519293    5837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519296    5837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519298    5837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519301    5837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519303    5837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519306    5837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519309    5837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519312    5837 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519314    5837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519317    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519321    5837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519324    5837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519328    5837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519331    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519335    5837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519338    5837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:36:51.530253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519342    5837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519346    5837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519349    5837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519352    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519354    5837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519363    5837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519366    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519369    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519371    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519374    5837 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519376    5837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519378    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519381    5837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519384    5837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519386    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519429    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519435    5837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519437    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519440    5837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:36:51.530813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519442    5837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519445    5837 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519448    5837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519451    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519453    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519456    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519458    5837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519461    5837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519463    5837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519466    5837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519468    5837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519471    5837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519473    5837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519475    5837 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519478    5837 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519481    5837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519483    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519486    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519488    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519498    5837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:36:51.531281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519500    5837 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519503    5837 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519505    5837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519508    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519510    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519513    5837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519515    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519517    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519520    5837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519523    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519525    5837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519527    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519530    5837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519532    5837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519535    5837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519538    5837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519541    5837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519543    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519545    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519548    5837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:36:51.531793 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519550    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519553    5837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519556    5837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519558    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519560    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519563    5837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519565    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519568    5837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519571    5837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519573    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519583    5837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519586    5837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519593    5837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519596    5837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519598    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519601    5837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519603    5837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519606    5837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519608    5837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:36:51.532279 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519611    5837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519613    5837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519616    5837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.519618    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.519650    5837 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521428    5837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521443    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521448    5837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521451    5837 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521454    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521457    5837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521460    5837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521463    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521466    5837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521468    5837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521471    5837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:36:51.532766 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521473    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521476    5837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521478    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521481    5837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521484    5837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521487    5837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521489    5837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521492    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521496    5837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521499    5837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521501    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521504    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521506    5837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521509    5837 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521512    5837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521515    5837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521518    5837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521521    5837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521523    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521526    5837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:36:51.533269 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521528    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521531    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521534    5837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521537    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521540    5837 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521543    5837 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521546    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521549    5837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521552    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521554    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521557    5837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521560    5837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521562    5837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521564    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521567    5837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521570    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521572    5837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521575    5837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521577    5837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521587    5837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:36:51.533914 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521590    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521592    5837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521595    5837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521597    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521600    5837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521603    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521606    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521609    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521611    5837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521615    5837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521620    5837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521622    5837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521625    5837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521628    5837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521630    5837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521632    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521635    5837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521638    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521640    5837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:36:51.534587 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521643    5837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521645    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521648    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521651    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521653    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521656    5837 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521658    5837 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521661    5837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521663    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521666    5837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521668    5837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521671    5837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521674    5837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521682    5837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521685    5837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521688    5837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521690    5837 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521693    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521695    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:36:51.535239 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521698    5837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521701    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521704    5837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521706    5837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521713    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521717    5837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521721    5837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521724    5837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521727    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521729    5837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521732    5837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521735    5837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521738    5837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521740    5837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521742    5837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521745    5837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:36:51.535916 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521748    5837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.521780    5837 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521818    5837 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521822    5837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521825    5837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521828    5837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521830    5837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521833    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521835    5837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521838    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521840    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521849    5837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521852    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521854    5837 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521857    5837 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521860    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521863    5837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:36:51.536371 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521866    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521868    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521870    5837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521873    5837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521877    5837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521880    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521882    5837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521885    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521887    5837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521890    5837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521892    5837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521895    5837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521897    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521900    5837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521902    5837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521905    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521907    5837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521910    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521912    5837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:36:51.537023 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521915    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521918    5837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521921    5837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521925    5837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521929    5837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521932    5837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521935    5837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521937    5837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521945    5837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521948    5837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521951    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521953    5837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521956    5837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521958    5837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521961    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521963    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521966    5837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521970    5837 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521972    5837 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:36:51.537667 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521975    5837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521977    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521980    5837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521982    5837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521985    5837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521987    5837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521990    5837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521992    5837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521995    5837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521997    5837 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.521999    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522002    5837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522005    5837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522007    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522010    5837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522012    5837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522015    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522018    5837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522022    5837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:36:51.538495 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522025    5837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522028    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522030    5837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522038    5837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522041    5837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522043    5837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522046    5837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522048    5837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522051    5837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522053    5837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522056    5837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522060    5837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522063    5837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522066    5837 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522069    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522071    5837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522074    5837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522077    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522079    5837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522082    5837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:36:51.539050 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522084    5837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522087    5837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522089    5837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522092    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522094    5837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522097    5837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522099    5837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522102    5837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522105    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522107    5837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522110    5837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522112    5837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522115    5837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:36:51.522117    5837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.522183    5837 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.523233    5837 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.523294    5837 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.523514    5837 server.go:1014] "Starting client certificate rotation"
Oct 30 12:36:51.539618 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.523647    5837 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:36:51.540057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.523707    5837 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:30:41 +0000 UTC" deadline="2028-06-14 03:58:12.410973929 +0000 UTC"
Oct 30 12:36:51.540057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.523742    5837 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14223h21m20.887234046s"
Oct 30 12:36:51.540057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.523968    5837 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:36:51.540057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.525160    5837 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:36:51.540057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.526647    5837 log.go:25] "Validated CRI v1 runtime API"
Oct 30 12:36:51.540057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.528671    5837 log.go:25] "Validated CRI v1 image API"
Oct 30 12:36:51.540057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.529434    5837 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 12:36:51.540057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.531951    5837 fs.go:135] Filesystem UUIDs: map[4f9c305b-8994-4260-987b-17215f91db9a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d8bc8d32-21b6-4c15-92a6-db6946fbc4cb:/dev/nvme0n1p3]
Oct 30 12:36:51.540291 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.531973    5837 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0855e4c7631992f259706fb48208b75d9b90d1740b84731caa06f74f63646877/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0855e4c7631992f259706fb48208b75d9b90d1740b84731caa06f74f63646877/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e3f71bc99acfb4c14662813364519fafe4f46385b1fbf306688db96514f26ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e3f71bc99acfb4c14662813364519fafe4f46385b1fbf306688db96514f26ca/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10f0c2bc66ec5fbc7ccbaab78b5d9724e568ca1d340056f0243bf684f86f97f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10f0c2bc66ec5fbc7ccbaab78b5d9724e568ca1d340056f0243bf684f86f97f0/userdata/shm major:0 minor:248 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32aa1f56c8be8accf7c779bf4f78e54aa34b11c7b8f4f503d9eaf693fb991370/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32aa1f56c8be8accf7c779bf4f78e54aa34b11c7b8f4f503d9eaf693fb991370/userdata/shm major:0 minor:277 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33263084b8c89445ef7119f0efd14350fcdd44baf9a2bff14ba6ded27d2e6685/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33263084b8c89445ef7119f0efd14350fcdd44baf9a2bff14ba6ded27d2e6685/userdata/shm major:0 minor:254 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42d7c5b6e1a1b85ce656693813ea8f7eb392ce931e9a4e06a49a0d75bbefbfd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42d7c5b6e1a1b85ce656693813ea8f7eb392ce931e9a4e06a49a0d75bbefbfd8/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45356e460e3e75b3bb9e7bd3d0049c5c050a496174f25db8d7bd3791152318e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45356e460e3e75b3bb9e7bd3d0049c5c050a496174f25db8d7bd3791152318e1/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48a7174dbcfa7e66136bd9bb60e499286f2f73f7cc0926589caade73ef8a917b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48a7174dbcfa7e66136bd9bb60e499286f2f73f7cc0926589caade73ef8a917b/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56079d226920cb26fe205a7aacbc2f3caa3e27655d3ca5172c0791fe7e354637/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56079d226920cb26fe205a7aacbc2f3caa3e27655d3ca5172c0791fe7e354637/userdata/shm major:0 minor:475 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/560843455db2355b850de37a5d81ae51e5429266263244dfb720309e6095a59c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/560843455db2355b850de37a5d81ae51e5429266263244dfb720309e6095a59c/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d4d1df1301def3a46aa74074625f528eebf276bef4a26e3e8cce05397f6189b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d4d1df1301def3a46aa74074625f528eebf276bef4a26e3e8cce05397f6189b/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82b27b3ae3931467c782820e573951472ce7db04f7d88f93702fb9a69fb06a00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82b27b3ae3931467c782820e573951472ce7db04f7d88f93702fb9a69fb06a00/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88f7490ae2fbb52873e01de2aa61d45b5d11f1f22e6ec8849f9a9d432763195e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88f7490ae2fbb52873e01de2aa61d45b5d11f1f22e6ec8849f9a9d432763195e/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8dbf2415cb6007a1bae0b4031c5d6363142904eb664535c9a1483fb13a3a3227/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8dbf2415cb6007a1bae0b4031c5d6363142904eb664535c9a1483fb13a3a3227/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9021c0a7ce94ad61b563080fc923ef0bb59486bf0102408a9c0969d70222e5cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9021c0a7ce94ad61b563080fc923ef0bb59486bf0102408a9c0969d70222e5cd/userdata/shm major:0 minor:482 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2917b8ca3190dbc570c0e7e783970b40fe6da05ddb92bfc02d3f55bb1a2d67e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2917b8ca3190dbc570c0e7e783970b40fe6da05ddb92bfc02d3f55bb1a2d67e/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a98e62006387c24317a806ce2becb38889c5ee81d1d2c62af2f8a74ff8d0bc14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a98e62006387c24317a806ce2becb38889c5ee81d1d2c62af2f8a74ff8d0bc14/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0ddff1458fcd7794a111b51a4622cf59e3f3a08b52c2cf78ea14f6adbae9e64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0ddff1458fcd7794a111b51a4622cf59e3f3a08b52c2cf78ea14f6adbae9e64/userdata/shm major:0 minor:438 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b11917bdda887b71d1dbf6a36b422e3b72951d800670d20ac351274ea1216dbd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b11917bdda887b71d1dbf6a36b422e3b72951d800670d20ac351274ea1216dbd/userdata/shm major:0 minor:250 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3397153b6c8da955283cddb4d3ba179f8ee3d9d90e55adb21e2dc34f94c554c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3397153b6c8da955283cddb4d3ba179f8ee3d9d90e55adb21e2dc34f94c554c/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b74e6a2dbfaaf708abc9916c24f331f1b9a7b0a6f7e675af08053d564ddbacd3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b74e6a2dbfaaf708abc9916c24f331f1b9a7b0a6f7e675af08053d564ddbacd3/userdata/shm major:0 minor:477 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b753ee78655981eb196701892fba5cdbb928afa5af679fdaf272a8bb977b3300/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b753ee78655981eb196701892fba5cdbb928afa5af679fdaf272a8bb977b3300/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1699a8cb635ca88e61742e21af223617c6801b8f9d66639db3e0a89af6b497b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1699a8cb635ca88e61742e21af223617c6801b8f9d66639db3e0a89af6b497b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e61575334e78a38be03a8b23bd58bc7c4937da4b4554f8bb54869a7768284963/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e61575334e78a38be03a8b23bd58bc7c4937da4b4554f8bb54869a7768284963/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a195064-4c82-47ec-939e-8f0ac45fb0f6/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0a195064-4c82-47ec-939e-8f0ac45fb0f6/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a195064-4c82-47ec-939e-8f0ac45fb0f6/volumes/kubernetes.io~projected/kube-api-access-h8gn6:{mountpoint:/var/lib/kubelet/pods/0a195064-4c82-47ec-939e-8f0ac45fb0f6/volumes/kubernetes.io~projected/kube-api-access-h8gn6 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a195064-4c82-47ec-939e-8f0ac45fb0f6/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0a195064-4c82-47ec-939e-8f0ac45fb0f6/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volume-subpaths/web-config/config-reloader/2 major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volume-subpaths/web-config/prometheus/15 major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~empty-dir/config-out major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~projected/kube-api-access-d57bx:{mountpoint:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~projected/kube-api-access-d57bx major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~projected/tls-assets major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/config major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:433 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/web-config major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14b22d89-19e4-4dbb-a031-91b5fe337785/volumes/kubernetes.io~projected/kube-api-access-774wq:{mountpoint:/var/lib/kubelet/pods/14b22d89-19e4-4dbb-a031-91b5fe337785/volumes/kubernetes.io~projected/kube-api-access-774wq major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/233c3bd3-01bc-4c71-8bc4-611933e34539/volumes/kubernetes.io~projected/kube-api-access-v766r:{mountpoint:/var/lib/kubelet/pods/233c3bd3-01bc-4c71-8bc4-611933e34539/volumes/kubernetes.io~projected/kube-api-access-v766r major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2343d79d-4146-4f1f-90be-ad84a256a4a5/volumes/kubernetes.io~projected/kube-api-access-tbhd4:{mountpoint:/var/lib/kubelet/pods/2343d79d-4146-4f1f-90be-ad84a256a4a5/volumes/kubernetes.io~projected/kube-api-access-tbhd4 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2343d79d-4146-4f1f-90be-ad84a256a4a5/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/2343d79d-4146-4f1f-90be-ad84a256a4a5/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30a15bca-3504-49fe-9ecf-c9860108905e/volumes/kubernetes.io~projected/kube-api-access-mtpt8:{mountpoint:/var/lib/kubelet/pods/30a15bca-3504-49fe-9ecf-c9860108905e/volumes/kubernetes.io~projected/kube-api-access-mtpt8 major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30a15bca-3504-49fe-9ecf-c9860108905e/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/30a15bca-3504-49fe-9ecf-c9860108905e/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70b50531-ed83-4bac-bdf1-0992042e01a0/volumes/kubernetes.io~projected/kube-api-access-xqhpx:{mountpoint:/var/lib/kubelet/pods/70b50531-ed83-4bac-bdf1-0992042e01a0/volumes/kubernetes.io~projected/kube-api-access-xqhpx major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70b50531-ed83-4bac-bdf1-0992042e01a0/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/70b50531-ed83-4bac-bdf1-0992042e01a0/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70b50531-ed83-4bac-bdf1-0992042e01a0/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/70b50531-ed83-4bac-bdf1-0992042e01a0/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7709b8a5-ca78-45d6-9c8c-136a13f50c72/volumes/kubernetes.io~projected/kube-api-access-8l6nw:{mountpoint:/var/lib/kubelet/pods/7709b8a5-ca78-45d6-9c8c-136a13f50c72/volumes/kubernetes.io~projected/kube-api-access-8l6nw major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7709b8a5-ca78-45d6-9c8c-136a13f50c72/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/7709b8a5-ca78-45d6-9c8c-136a13f50c72/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7709b8a5-ca78-45d6-9c8c-136a13f50c72/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/7709b8a5-ca78-45d6-9c8c-136a13f50c72/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7709b8a5-ca78-45d6-9c8c-136a13f50c72/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/7709b8a5-ca78-45d6-9c8c-136a13f50c72/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volume-subpaths/web-config/alertmanager/9 major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volume-subpaths/web-config/config-reloader/6 major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~empty-dir/config-out major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~projected/kube-api-access-vqncb:{mountpoint:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~projected/kube-api-access-vqncb major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~projected/tls-assets major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/config-volume major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/web-config major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/996125c2-4148-45f2-845d-33e743fdd73a/volumes/kubernetes.io~projected/kube-api-access-xtx6n:{mountpoint:/var/lib/kubelet/pods/996125c2-4148-45f2-845d-33e743fdd73a/volumes/kubernetes.io~projected/kube-api-access-xtx6n major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b/volumes/kubernetes.io~projected/kube-api-access-255mf:{mountpoint:/var/lib/kubelet/pods/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b/volumes/kubernetes.io~projected/kube-api-access-255mf major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b/volumes/kubernetes.io~secret/metrics-certs major:0 minor:470 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~projected/kube-api-access-z8w42:{mountpoint:/var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~projected/kube-api-access-z8w42 major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef/volumes/kubernetes.io~projected/kube-api-access-lbg47:{mountpoint:/var/lib/kubelet/pods/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef/volumes/kubernetes.io~projected/kube-api-access-lbg47 major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbb54d63-c3cc-4759-ba36-252e1ffc4d18/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/bbb54d63-c3cc-4759-ba36-252e1ffc4d18/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c72d4793-688e-4c37-bb51-e0a9100a2203/volumes/kubernetes.io~projected/kube-api-access-h77bx:{mountpoint:/var/lib/kubelet/pods/c72d4793-688e-4c37-bb51-e0a9100a2203/volumes/kubernetes.io~projected/kube-api-access-h77bx major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca860770-e681-4dae-806f-ef6de17bed8e/volumes/kubernetes.io~projected/kube-api-access-74n4f:{mountpoint:/var/lib/kubelet/pods/ca860770-e681-4dae-806f-ef6de17bed8e/volumes/kubernetes.io~projected/kube-api-access-74n4f major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd722809-68b9-41b8-8a5e-814903a74299/volumes/kubernetes.io~projected/kube-api-access-fhm4h:{mountpoint:/var/lib/kubelet/pods/cd722809-68b9-41b8-8a5e-814903a74299/volumes/kubernetes.io~projected/kube-api-access-fhm4h major:0 minor:479 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddd38ac5-1413-4345-9452-05936c2b31b8/volumes/kubernetes.io~projected/kube-api-access-ndn6b:{mountpoint:/var/lib/kubelet/pods/ddd38ac5-1413-4345-9452-05936c2b31b8/volumes/kubernetes.io~projected/kube-api-access-ndn6b major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddd38ac5-1413-4345-9452-05936c2b31b8/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/ddd38ac5-1413-4345-9452-05936c2b31b8/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e383bb23-4967-47e8-afdf-937f840aea60/volumes/kubernetes.io~projected/kube-api-access-nkqw9:{mountpoint:/var/lib/kubelet/pods/e383bb23-4967-47e8-afdf-937f840aea60/volumes/kubernetes.io~projected/kube-api-access-nkqw9 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e383bb23-4967-47e8-afdf-937f840aea60/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/e383bb23-4967-47e8-afdf-937f840aea60/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4fe0229-c475-400a-b6c0-48d80476b3b5/volumes/kubernetes.io~projected/kube-api-access-rfhl5:{mountpoint:/var/lib/kubelet/pods/e4fe0229-c475-400a-b6c0-48d80476b3b5/volumes/kubernetes.io~projected/kube-api-access-rfhl5 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb31cffb-32c0-4d28-9947-325621908909/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/eb31cffb-32c0-4d28-9947-325621908909/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb31cffb-32c0-4d28-9947-325621908909/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/eb31cffb-32c0-4d28-9947-325621908909/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb31cffb-32c0-4d28-9947-325621908909/volumes/kubernetes.io~projected/kube-api-access-c24sf:{mountpoint:/var/lib/kubelet/pods/eb31cffb-32c0-4d28-9947-325621908909/volumes/kubernetes.io~projected/kube-api-access-c24sf major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7ad82f0-86a1-43e3-8237-846a3aef0c88/volumes/kubernetes.io~projected/kube-api-access-gc6np:{mountpoint:/var/lib/kubelet/pods/f7ad82f0-86a1-43e3-8237-846a3aef0c88/volumes/kubernetes.io~projected/kube-api-access-gc6np major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7ad82f0-86a1-43e3-8237-846a3aef0c88/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f7ad82f0-86a1-43e3-8237-846a3aef0c88/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7ad82f0-86a1-43e3-8237-846a3aef0c88/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/f7ad82f0-86a1-43e3-8237-846a3aef0c88/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:245 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4dca68f47e0d4e6123960645f59ae6c88ebd94f05e10a0f85be787cdde4af702/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4a27b680a6a6a7265ece4db5f60cec9e0ac5ff6b1396dc5803c49bf63e2b57f6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/53f0198fc27a3d66be9745af8b2b63c3cd78e2ebac60d307e993656d58b9eec4/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/04e262dcc57b586e80263e8c543a1ee56e2f1ada8269690ec9e63ed9709ffed7/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/136060c26fbf0bec374cd813d11790b8a965e196c94465f794a7388690a8a625/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/730c2fd5a8c12fd49d5e3744bf33fe5097dd0b84cf0c2b255c2c715fcc68e63c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/463c59b526131ca0b25c56a36d11ef40da95eca7b9005e28cf9132e1fa4ea315/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/964951a5b50f6da5190eb169f188aa262a5cc4e5700ab60327b36fa273988fef/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/14af6b09bfa58f8b92f6bc9e1f02480aa38e6e9eaf6806cf1adcdf4845069d87/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/f07efa9cb144dbf3b940e4b3a70cd67b67cc0c535941271156a736393458c2e6/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/dffc14d47b22e06e0872d550e5e7e5556aa42418ca06cd09b788eae5692574ac/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/baba001c43acc9ff46a72b65cd794706377b89f426a6bbc01e5bac8f7786b776/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ef8f3761df73dc91a0a85f2f094c3b86609b3d0d15662ff3375ffb233a1bc613/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/8de594a0dea194083ce481396ec672d58724575482356698feee48a012986974/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-156:{mountpoint:/var/lib/containers/storage/overlay/39655b5257546bb671ca51c30bd9babec643a743cf7c2d6928d3728a2dbe1369/merged major:0 minor:156 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d4df1ba8f44ccf3d42df1a311bdec1ac9d927642a64e5a6546885877fe9eab46/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e75ffbaeb12c465ea9d747ac83c57d33dafa92f3f46bf6d0f13a939d963b45d0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9b9262f17b8cee34aa245530ab2abac8adb28b37a6f95de694df75d97690b801/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d4fb6dbb30643ae0b5e94756621d40cafa2acbf6c85e9f6b9e71ce058646196d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/073564166a42a36ba6a89da332c146c731e5e7b5948459d063b57ec6d1141c0a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4d55b2bbe776842971054c95b5744c55d5a083957184b8da6032e0247e98188d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/723d77850c945345b931e782324e1db4be8068deaa1e3bfb74f3247b454abb87/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/2b66a50e0a5844f4b2117dce396899967c1694af0234ffe2bb62e4f48a3efb37/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/1b2a56924f732cce47dc080d35bdc022e829f7c68075a861d8de55139f958582/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/28bfb76a329dd13e6195854f6a7861e25aa65aa253ff73f08f21a66ed13657d5/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/837e90be14084ea45bc539aff7037253051209e30cb342945d281a2ca9220cff/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/b799214bcdcc9910e8878baa613c6ec2e3cffecc00ef739465ec61a8a2cc5f81/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/49b15493cbfc7a420964a15428d200028c5e60aae973c582d97b684317ff9f09/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/217ccb6d27efa1b8cdd2252fec3aee344a74b14b705dfa34104818ad77b1e2f4/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/3ff51cdc7605df4a7d060189110c5ea851486144483a947a73242e4edfcc88b3/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/f9324ca1b4585f3d1c7b7a6bf9d114f06cd6e593829d47f3873a6cf367602e3d/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/1846156891c68a89ca6ef597ec306619c29d3bd6ae110a445726f4310b0572e7/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/6d90ea6285560cd79942d0726898a22e43084fce2cbb464a63ae67f4e1645883/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/6a2c1854b19215c34d6f85cc4a6100b149f66e978f78da312889106aa3809dc1/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/2ba291f751cf085b3c564131c941838a8f219e9953ed429ad947a2409d656fd0/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/43500279a27330d031f5b200c6b71b60d8519d3756d4f10a8aeeef6ff9efde6c/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/2e10593a3f2ddd8a22e30513f16af3a6ad6fc577d320da2e4bb4673b9e59782f/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/2a966cbaad90dc4f0dbd590cb6ea16c37c6365313d095cad9759d8b6c78a0ff6/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/b05e0cc665d2049eda649af91c1827191ce2bf6c409e26e0f2c63a5fa01ae063/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/64d25341ddd04f960a0941863457e392d1cfae6aa2f14b8e8af332a931f84ff7/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/f9cf3630d7faf7dc820b254926bd598d35a39a9d19bdc1c779c36a3c48a54e86/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/8e9a1771de48df90a11376d3d0a6aba4051f093432cf61defb6cc935553513e3/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/a1413ec3f180a461e7cf5c5dc512bfebd45db845c6e4940bfc27df4caa710137/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/0ec44d2bf49650635232553110f4cd93a575a62492003fb87ab25b060174e3fa/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/2831f1de9e62c09757adde867e6c6c616501614a7d3e63e8d80837e2ca524bac/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/9b626b8e815205ec1fc16390d9c0523d24908df17248c31462aa6f8aa415ec96/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/3883416a93630af27790db1cafce84c06f36f0b252e0ab368884cc87b8cde9a5/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/faefd191ce8b8548f6b2a8018c2186c97830a1cea2f84fa1f4bef826851ed92d/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/2f2e808c8c9a87429ab0b252f9143f7415a5476694aa71bf0e992fde2fd50a8e/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/5c57c2ec65ee01b3ac8ef2c712dbf3e8d8742f36b32f0e3d3c3fed264434aead/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/9240bace052eabbac261d0a3fb01297fbdbb221a18314fb5d24f8a655f54205e/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/3dac7ae413f8b31587b6776e94188d400131667e7d517c56cb7538872f1b15d8/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/627d086442a18734af621e2716b57f91c0b01a9d37391e9986cc17286a248f7e/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/1a192bb250134e6620486e54ae1c97fbee2eea4b1871e8869cd516a910105983/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/72cb458e3e6eac57a15732e876376eb4c325a09b9c9c0b9908a664b221561d81/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/3a4e2658f748c0f4b7e9886c602f0c723b9075673d0f34ac4c31ab1994d1a527/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/a207f6f4c0b5123a4668cf1b78fe004df777445c898172af0ca46a6f3be78c27/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/975f9a741836bae9e545d7a6135a43c82ff9c9aaecc0fa64256425c1093d7348/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/172ade1155e47289858d7a76d0bb75a76e17118ff6d5af4b8b58ab7babe507a6/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/6ebc091e6f5d7c4846b7a404f75e30a02913e9efda1e9557bb03d23127affdfe/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/e1f113787033c21395b58094a65e85cabb3c1879a5f88ae0bbb107caf370d589/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/269ec406b19ecd14377cf8b2ca44d6c687a46feb73dfd892246ea2d3ac50fd3e/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/2c1af8fb3e6f2214c6e0a1ec9ddc1bc4f1ec32a7f49c025cd2e160b329db4e1d/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/32300e359c508b820e8670b43128f08cae7d999bec3dc2223fc52b9bcd5682dd/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/d6ad9aadcc6872ffe0f1c5a48d73fc9da767d488c621d881911fbcae83e9178f/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-465:{mountpoint:/var/lib/containers/storage/overlay/fe3c556e4fefdc33fcd7453e833736721afa50ecd2b88a5fabacd20e7cc2983f/merged major:0 minor:465 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/e484911a05d4f2c793b514c5996c10a469abcbc0225be727015b6aaf8bdaf631/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/aff868ce40159a2f6c536a7ee41b358aa65561d4b5c823d269759bac60d8b72a/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/8d0e97e24ddbdbbbe486040197b34e43878b7b4685d6dd0c0d91476818a0b868/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/b839fa003f838912f75b6f3284d23b780dcc6b4488b6167a9806a552074bca4a/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-490:{mountpoint:/var/lib/containers/storage/overlay/e019789ee0c8686d5c9b1c01b43849a7aa93ca463b12ba71084e4c99bc4494de/merged major:0 minor:490 fsType:overlay blockSize:0} overlay_0-496:{mountpoint:/var/lib/containers/storage/overlay/830d3bb3140080e5dafe1fe422a7d5818658c20cd010ad631324de2667dcca5d/merged major:0 minor:496 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/0a1ce73a5e59dad7515970b889f7927ad45d200439fa6d11df27ce8cfaf32f31/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/82cef8ecf499b01bad4250186bcfe794775fc1bdf68a7a22d02eaa9f11989640/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f311e8a10388b3e793238e0e6072e9314d419932c1a658bb7ad8efaa7000f91b/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ef9236a8d7a9502949fdedf5b824071a70dd373d53ecb72da81a8301c70b5071/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/654b51d40f641814b2c3d46cb9083229653913aa32e62e6a9791aa610f48f4a5/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/a1ff914262a57f08c62255bfec606ab870e8202795f0af482540e7d69d78f0ca/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/8d19b2a2c521b927136b58d65dddc4afb03f5e13fff4e3abf5a8b4388f670b96/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/36d6adeb63d594f8c13ef7575389ef89575c308849dad9075776b204fb39ad96/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/0c6b9a1b942e69e9166f78a3aee09869c874a135e02f2770bbc9b727b97250bf/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/fb7beddb22b8294c93b1b4da42e184c72c514fcc120c14adf09da6f9367dd11a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a632e2d12a71e163dc8e17d1c352dc0b6ffa545e8f74d479e94f7264c0a71210/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 12:36:51.541861 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.541378    5837 manager.go:217] Machine: {Timestamp:2026-10-30 12:36:51.540873899 +0000 UTC m=+0.065158127 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100515 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25b0a17f88fcfef32e0bd984b33381 SystemUUID:ec25b0a1-7f88-fcfe-f32e-0bd984b33381 BootID:e1c59c65-9035-4538-a947-a0192302a5c6 Filesystems:[{Device:/run/containers/storage/overlay-containers/b74e6a2dbfaaf708abc9916c24f331f1b9a7b0a6f7e675af08053d564ddbacd3/userdata/shm DeviceMajor:0 DeviceMinor:477 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:283 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/eb31cffb-32c0-4d28-9947-325621908909/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:240 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7709b8a5-ca78-45d6-9c8c-136a13f50c72/volumes/kubernetes.io~projected/kube-api-access-8l6nw DeviceMajor:0 DeviceMinor:341 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:422 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:432 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:470 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c72d4793-688e-4c37-bb51-e0a9100a2203/volumes/kubernetes.io~projected/kube-api-access-h77bx DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:278 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:434 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8dbf2415cb6007a1bae0b4031c5d6363142904eb664535c9a1483fb13a3a3227/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-156 DeviceMajor:0 DeviceMinor:156 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a195064-4c82-47ec-939e-8f0ac45fb0f6/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4fe0229-c475-400a-b6c0-48d80476b3b5/volumes/kubernetes.io~projected/kube-api-access-rfhl5 DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7ad82f0-86a1-43e3-8237-846a3aef0c88/volumes/kubernetes.io~projected/kube-api-access-gc6np DeviceMajor:0 DeviceMinor:246 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:290 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82b27b3ae3931467c782820e573951472ce7db04f7d88f93702fb9a69fb06a00/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:427 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b753ee78655981eb196701892fba5cdbb928afa5af679fdaf272a8bb977b3300/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef/volumes/kubernetes.io~projected/kube-api-access-lbg47 DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f7ad82f0-86a1-43e3-8237-846a3aef0c88/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:242 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/14b22d89-19e4-4dbb-a031-91b5fe337785/volumes/kubernetes.io~projected/kube-api-access-774wq DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e383bb23-4967-47e8-afdf-937f840aea60/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e383bb23-4967-47e8-afdf-937f840aea60/volumes/kubernetes.io~projected/kube-api-access-nkqw9 DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:278 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e3f71bc99acfb4c14662813364519fafe4f46385b1fbf306688db96514f26ca/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/eb31cffb-32c0-4d28-9947-325621908909/volumes/kubernetes.io~projected/kube-api-access-c24sf DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42d7c5b6e1a1b85ce656693813ea8f7eb392ce931e9a4e06a49a0d75bbefbfd8/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:278 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0ddff1458fcd7794a111b51a4622cf59e3f3a08b52c2cf78ea14f6adbae9e64/userdata/shm DeviceMajor:0 DeviceMinor:438 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/560843455db2355b850de37a5d81ae51e5429266263244dfb720309e6095a59c/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:324 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:433 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7709b8a5-ca78-45d6-9c8c-136a13f50c72/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:340 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:285 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9021c0a7ce94ad61b563080fc923ef0bb59486bf0102408a9c0969d70222e5cd/userdata/shm DeviceMajor:0 DeviceMinor:482 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3397153b6c8da955283cddb4d3ba179f8ee3d9d90e55adb21e2dc34f94c554c/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70b50531-ed83-4bac-bdf1-0992042e01a0/volumes/kubernetes.io~projected/kube-api-access-xqhpx DeviceMajor:0 DeviceMinor:247 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e61575334e78a38be03a8b23bd58bc7c4937da4b4554f8bb54869a7768284963/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70b50531-ed83-4bac-bdf1-0992042e01a0/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:244 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:436 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b/volumes/kubernetes.io~projected/kube-api-access-255mf DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:329 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/eb31cffb-32c0-4d28-9947-325621908909/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ca860770-e681-4dae-806f-ef6de17bed8e/volumes/kubernetes.io~projected/kube-api-access-74n4f DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88f7490ae2fbb52873e01de2aa61d45b5d11f1f22e6ec8849f9a9d432763195e/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2343d79d-4146-4f1f-90be-ad84a256a4a5/volumes/kubernetes.io~projected/kube-api-access-tbhd4 DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:435 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddd38ac5-1413-4345-9452-05936c2b31b8/volumes/kubernetes.io~projected/kube-api-access-ndn6b DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0855e4c7631992f259706fb48208b75d9b90d1740b84731caa06f74f63646877/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:284 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56079d226920cb26fe205a7aacbc2f3caa3e27655d3ca5172c0791fe7e354637/userdata/shm DeviceMajor:0 DeviceMinor:475 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/0a195064-4c82-47ec-939e-8f0ac45fb0f6/volumes/kubernetes.io~projected/kube-api-access-h8gn6 DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a98e62006387c24317a806ce2becb38889c5ee81d1d2c62af2f8a74ff8d0bc14/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ddd38ac5-1413-4345-9452-05936c2b31b8/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:285 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~projected/kube-api-access-vqncb DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48a7174dbcfa7e66136bd9bb60e499286f2f73f7cc0926589caade73ef8a917b/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d4d1df1301def3a46aa74074625f528eebf276bef4a26e3e8cce05397f6189b/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:434 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:289 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7709b8a5-ca78-45d6-9c8c-136a13f50c72/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:339 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1699a8cb635ca88e61742e21af223617c6801b8f9d66639db3e0a89af6b497b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2343d79d-4146-4f1f-90be-ad84a256a4a5/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7ad82f0-86a1-43e3-8237-846a3aef0c88/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:245 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:330 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b11917bdda887b71d1dbf6a36b422e3b72951d800670d20ac351274ea1216dbd/userdata/shm DeviceMajor:0 DeviceMinor:250 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:428 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:429 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:431 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45356e460e3e75b3bb9e7bd3d0049c5c050a496174f25db8d7bd3791152318e1/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:286 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30a15bca-3504-49fe-9ecf-c9860108905e/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:351 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:434 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd722809-68b9-41b8-8a5e-814903a74299/volumes/kubernetes.io~projected/kube-api-access-fhm4h DeviceMajor:0 DeviceMinor:479 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-490 DeviceMajor:0 DeviceMinor:490 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~projected/kube-api-access-d57bx DeviceMajor:0 DeviceMinor:437 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-465 DeviceMajor:0 DeviceMinor:465 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-496 DeviceMajor:0 DeviceMinor:496 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbb54d63-c3cc-4759-ba36-252e1ffc4d18/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:434 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:474 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:282 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~projected/kube-api-access-z8w42 DeviceMajor:0 DeviceMinor:333 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/996125c2-4148-45f2-845d-33e743fdd73a/volumes/kubernetes.io~projected/kube-api-access-xtx6n DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32aa1f56c8be8accf7c779bf4f78e54aa34b11c7b8f4f503d9eaf693fb991370/userdata/shm DeviceMajor:0 DeviceMinor:277 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:332 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7709b8a5-ca78-45d6-9c8c-136a13f50c72/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:338 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/30a15bca-3504-49fe-9ecf-c9860108905e/volumes/kubernetes.io~projected/kube-api-access-mtpt8 DeviceMajor:0 DeviceMinor:352 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ecfafe4-94c0-451e-b646-54aa9dce7f46/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:331 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:278 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70b50531-ed83-4bac-bdf1-0992042e01a0/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:243 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33263084b8c89445ef7119f0efd14350fcdd44baf9a2bff14ba6ded27d2e6685/userdata/shm DeviceMajor:0 DeviceMinor:254 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a195064-4c82-47ec-939e-8f0ac45fb0f6/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:241 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10f0c2bc66ec5fbc7ccbaab78b5d9724e568ca1d340056f0243bf684f86f97f0/userdata/shm DeviceMajor:0 DeviceMinor:248 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2917b8ca3190dbc570c0e7e783970b40fe6da05ddb92bfc02d3f55bb1a2d67e/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/233c3bd3-01bc-4c71-8bc4-611933e34539/volumes/kubernetes.io~projected/kube-api-access-v766r DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:430 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0855e4c7631992f MacAddress:2a:da:28:9b:db:29 Speed:10000 Mtu:8901} {Name:10f0c2bc66ec5fb MacAddress:52:93:36:cd:ab:5a Speed:10000 Mtu:8901} {Name:32aa1f56c8be8ac MacAddress:2a:4d:b2:7b:16:18 Speed:10000 Mtu:8901} {Name:33263084b8c8944 MacAddress:76:10:f7:d1:49:d0 Speed:10000 Mtu:8901} {Name:45356e460e3e75b MacAddress:f2:ee:50:69:73:76 Speed:10000 Mtu:8901} {Name:56079d226920cb2 MacAddress:32:b3:4a:78:cc:ea Speed:10000 Mtu:8901} {Name:5d4d1df1301def3 MacAddress:4a:b3:21:05:a4:ee Speed:10000 Mtu:8901} {Name:82b27b3ae393146 MacAddress:56:d9:08:c2:99:ba Speed:10000 Mtu:8901} {Name:9021c0a7ce94ad6 MacAddress:d2:c9:f0:d7:e1:f2 Speed:10000 Mtu:8901} {Name:a2917b8ca3190db MacAddress:22:17:6b:c6:21:51 Speed:10000 Mtu:8901} {Name:b0ddff1458fcd77 MacAddress:5e:9b:ee:92:cb:03 Speed:10000 Mtu:8901} {Name:b74e6a2dbfaaf70 MacAddress:b2:c3:4b:3b:a0:e0 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f3:ac:18:41 Speed:0 Mtu:9001} {Name:br-int MacAddress:7e:32:a9:26:ce:d3 Speed:0 Mtu:8901} {Name:e61575334e78a38 MacAddress:62:51:1e:e7:11:30 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f3:ac:18:41 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:96:64:f0:7f:e1:80 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:42:4b:4b:4c:e1:80 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 12:36:51.542111 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.541867    5837 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 12:36:51.542111 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.541920    5837 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 12:36:51.542202 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.542181    5837 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 12:36:51.542361 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.542203    5837 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-228.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 12:36:51.542413 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.542365    5837 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 12:36:51.542413 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.542373    5837 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 12:36:51.542413 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.542387    5837 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:36:51.542500 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.542418    5837 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:36:51.542569 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.542557    5837 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 12:36:51.542787 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.542777    5837 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 12:36:51.542968 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.542894    5837 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 12:36:51.542968 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.542932    5837 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 12:36:51.542968 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.542946    5837 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 12:36:51.542968 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.542960    5837 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 12:36:51.543136 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.542980    5837 kubelet.go:405] "Adding apiserver pod source"
Oct 30 12:36:51.543136 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.542991    5837 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 12:36:51.543919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.543901    5837 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 12:36:51.543919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.543916    5837 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:36:51.544531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.544509    5837 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 12:36:51.547233 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547214    5837 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 12:36:51.547233 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547229    5837 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:36:51.547379 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547347    5837 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 12:36:51.547631 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547615    5837 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 12:36:51.547691 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547656    5837 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 12:36:51.547782 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547770    5837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 12:36:51.547836 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547790    5837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 12:36:51.547836 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547799    5837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 12:36:51.547836 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547808    5837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 12:36:51.547836 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547817    5837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 12:36:51.547836 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547826    5837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 12:36:51.547836 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547834    5837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 12:36:51.548105 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547843    5837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 12:36:51.548105 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547853    5837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 12:36:51.548105 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547864    5837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 12:36:51.548105 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547887    5837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 12:36:51.548105 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547902    5837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 12:36:51.548105 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547935    5837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 12:36:51.548105 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.547945    5837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 12:36:51.551630 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.551613    5837 server.go:1263] "Started kubelet"
Oct 30 12:36:51.554147 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.551714    5837 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 12:36:51.554147 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.551764    5837 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 12:36:51.554147 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.551858    5837 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 12:36:51.554147 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.552292    5837 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 12:36:51.552193 ip-10-0-137-228 systemd[1]: Started Kubernetes Kubelet.
Oct 30 12:36:51.556973 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.555142    5837 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 12:36:51.559122 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.559104    5837 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 12:36:51.559216 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.559161    5837 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 12:36:51.559216 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.559173    5837 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 12:36:51.559216 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:36:51.559184    5837 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-228.ec2.internal\" not found"
Oct 30 12:36:51.559365 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.559271    5837 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.559851    5837 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.559936    5837 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:30:42 +0000 UTC" deadline="2028-05-30 02:32:42.89533739 +0000 UTC"
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.559997    5837 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13861h55m51.335344405s"
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562091    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddd38ac5-1413-4345-9452-05936c2b31b8" volumeName="kubernetes.io/secret/ddd38ac5-1413-4345-9452-05936c2b31b8-cert" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562135    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb31cffb-32c0-4d28-9947-325621908909" volumeName="kubernetes.io/projected/eb31cffb-32c0-4d28-9947-325621908909-kube-api-access-c24sf" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562151    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562167    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562181    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562206    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562219    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70b50531-ed83-4bac-bdf1-0992042e01a0" volumeName="kubernetes.io/projected/70b50531-ed83-4bac-bdf1-0992042e01a0-kube-api-access-xqhpx" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562232    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b16518-f7fa-4ddf-ab14-ace75b8d127b" volumeName="kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-web-config" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562247    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b16518-f7fa-4ddf-ab14-ace75b8d127b" volumeName="kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562260    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e383bb23-4967-47e8-afdf-937f840aea60" volumeName="kubernetes.io/empty-dir/e383bb23-4967-47e8-afdf-937f840aea60-data-volume" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562270    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14b22d89-19e4-4dbb-a031-91b5fe337785" volumeName="kubernetes.io/configmap/14b22d89-19e4-4dbb-a031-91b5fe337785-cni-binary-copy" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562277    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a195064-4c82-47ec-939e-8f0ac45fb0f6" volumeName="kubernetes.io/secret/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562284    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4" volumeName="kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562294    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14b22d89-19e4-4dbb-a031-91b5fe337785" volumeName="kubernetes.io/projected/14b22d89-19e4-4dbb-a031-91b5fe337785-kube-api-access-774wq" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562304    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2343d79d-4146-4f1f-90be-ad84a256a4a5" volumeName="kubernetes.io/configmap/2343d79d-4146-4f1f-90be-ad84a256a4a5-config-volume" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562313    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7709b8a5-ca78-45d6-9c8c-136a13f50c72" volumeName="kubernetes.io/projected/7709b8a5-ca78-45d6-9c8c-136a13f50c72-kube-api-access-8l6nw" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562320    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b16518-f7fa-4ddf-ab14-ace75b8d127b" volumeName="kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562326    5837 factory.go:55] Registering systemd factory
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562342    5837 factory.go:223] Registration of the systemd container factory successfully
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562329    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ecfafe4-94c0-451e-b646-54aa9dce7f46" volumeName="kubernetes.io/projected/9ecfafe4-94c0-451e-b646-54aa9dce7f46-kube-api-access-z8w42" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562421    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a195064-4c82-47ec-939e-8f0ac45fb0f6" volumeName="kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562444    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562457    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562467    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562479    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70b50531-ed83-4bac-bdf1-0992042e01a0" volumeName="kubernetes.io/secret/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562490    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b16518-f7fa-4ddf-ab14-ace75b8d127b" volumeName="kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-metrics-client-ca" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562504    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b16518-f7fa-4ddf-ab14-ace75b8d127b" volumeName="kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-volume" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562517    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ecfafe4-94c0-451e-b646-54aa9dce7f46" volumeName="kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562529    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ecfafe4-94c0-451e-b646-54aa9dce7f46" volumeName="kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562537    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ecfafe4-94c0-451e-b646-54aa9dce7f46" volumeName="kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562550    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef" volumeName="kubernetes.io/projected/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-kube-api-access-lbg47" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562558    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c72d4793-688e-4c37-bb51-e0a9100a2203" volumeName="kubernetes.io/configmap/c72d4793-688e-4c37-bb51-e0a9100a2203-serviceca" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562566    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4fe0229-c475-400a-b6c0-48d80476b3b5" volumeName="kubernetes.io/projected/e4fe0229-c475-400a-b6c0-48d80476b3b5-kube-api-access-rfhl5" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562574    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7ad82f0-86a1-43e3-8237-846a3aef0c88" volumeName="kubernetes.io/secret/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562582    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-tls-assets" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562591    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="233c3bd3-01bc-4c71-8bc4-611933e34539" volumeName="kubernetes.io/projected/233c3bd3-01bc-4c71-8bc4-611933e34539-kube-api-access-v766r" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562599    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70b50531-ed83-4bac-bdf1-0992042e01a0" volumeName="kubernetes.io/secret/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-tls" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562611    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b16518-f7fa-4ddf-ab14-ace75b8d127b" volumeName="kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562622    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b16518-f7fa-4ddf-ab14-ace75b8d127b" volumeName="kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-cluster-tls-config" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562631    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ecfafe4-94c0-451e-b646-54aa9dce7f46" volumeName="kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562639    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef" volumeName="kubernetes.io/configmap/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-metrics-client-ca" seLinuxMountContext=""
Oct 30 12:36:51.562600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562647    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb31cffb-32c0-4d28-9947-325621908909" volumeName="kubernetes.io/empty-dir/eb31cffb-32c0-4d28-9947-325621908909-tmp" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562656    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a195064-4c82-47ec-939e-8f0ac45fb0f6" volumeName="kubernetes.io/projected/0a195064-4c82-47ec-939e-8f0ac45fb0f6-kube-api-access-h8gn6" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562657    5837 factory.go:153] Registering CRI-O factory
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562676    5837 factory.go:223] Registration of the crio container factory successfully
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562743    5837 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562764    5837 factory.go:103] Registering Raw factory
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562780    5837 manager.go:1199] Started watching for new ooms in manager
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562664    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30a15bca-3504-49fe-9ecf-c9860108905e" volumeName="kubernetes.io/projected/30a15bca-3504-49fe-9ecf-c9860108905e-kube-api-access-mtpt8" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562872    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70b50531-ed83-4bac-bdf1-0992042e01a0" volumeName="kubernetes.io/configmap/70b50531-ed83-4bac-bdf1-0992042e01a0-metrics-client-ca" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562899    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef" volumeName="kubernetes.io/secret/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562916    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e383bb23-4967-47e8-afdf-937f840aea60" volumeName="kubernetes.io/configmap/e383bb23-4967-47e8-afdf-937f840aea60-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562930    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7ad82f0-86a1-43e3-8237-846a3aef0c88" volumeName="kubernetes.io/configmap/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562944    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7ad82f0-86a1-43e3-8237-846a3aef0c88" volumeName="kubernetes.io/projected/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-api-access-gc6np" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562958    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562975    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config-out" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.562989    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e383bb23-4967-47e8-afdf-937f840aea60" volumeName="kubernetes.io/secret/e383bb23-4967-47e8-afdf-937f840aea60-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563002    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a195064-4c82-47ec-939e-8f0ac45fb0f6" volumeName="kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-env-overrides" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563016    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="233c3bd3-01bc-4c71-8bc4-611933e34539" volumeName="kubernetes.io/configmap/233c3bd3-01bc-4c71-8bc4-611933e34539-iptables-alerter-script" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563030    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70b50531-ed83-4bac-bdf1-0992042e01a0" volumeName="kubernetes.io/empty-dir/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-textfile" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563044    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7709b8a5-ca78-45d6-9c8c-136a13f50c72" volumeName="kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563058    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b16518-f7fa-4ddf-ab14-ace75b8d127b" volumeName="kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563073    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ecfafe4-94c0-451e-b646-54aa9dce7f46" volumeName="kubernetes.io/configmap/9ecfafe4-94c0-451e-b646-54aa9dce7f46-metrics-client-ca" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563099    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e383bb23-4967-47e8-afdf-937f840aea60" volumeName="kubernetes.io/projected/e383bb23-4967-47e8-afdf-937f840aea60-kube-api-access-nkqw9" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563113    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb31cffb-32c0-4d28-9947-325621908909" volumeName="kubernetes.io/empty-dir/eb31cffb-32c0-4d28-9947-325621908909-etc-tuned" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563121    5837 manager.go:319] Starting recovery of all containers
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563126    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14b22d89-19e4-4dbb-a031-91b5fe337785" volumeName="kubernetes.io/configmap/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-daemon-config" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563141    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2343d79d-4146-4f1f-90be-ad84a256a4a5" volumeName="kubernetes.io/secret/2343d79d-4146-4f1f-90be-ad84a256a4a5-metrics-tls" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563155    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7709b8a5-ca78-45d6-9c8c-136a13f50c72" volumeName="kubernetes.io/configmap/7709b8a5-ca78-45d6-9c8c-136a13f50c72-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563169    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="996125c2-4148-45f2-845d-33e743fdd73a" volumeName="kubernetes.io/projected/996125c2-4148-45f2-845d-33e743fdd73a-kube-api-access-xtx6n" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563182    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b" volumeName="kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563195    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca860770-e681-4dae-806f-ef6de17bed8e" volumeName="kubernetes.io/projected/ca860770-e681-4dae-806f-ef6de17bed8e-kube-api-access-74n4f" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563209    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563222    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-web-config" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563235    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7709b8a5-ca78-45d6-9c8c-136a13f50c72" volumeName="kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-client-ca-bundle" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563250    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b16518-f7fa-4ddf-ab14-ace75b8d127b" volumeName="kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-kube-api-access-vqncb" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563267    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="996125c2-4148-45f2-845d-33e743fdd73a" volumeName="kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563280    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbb54d63-c3cc-4759-ba36-252e1ffc4d18" volumeName="kubernetes.io/secret/bbb54d63-c3cc-4759-ba36-252e1ffc4d18-agent-certs" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563293    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4fe0229-c475-400a-b6c0-48d80476b3b5" volumeName="kubernetes.io/empty-dir/e4fe0229-c475-400a-b6c0-48d80476b3b5-tmp-dir" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563307    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7ad82f0-86a1-43e3-8237-846a3aef0c88" volumeName="kubernetes.io/secret/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563322    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563349    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7709b8a5-ca78-45d6-9c8c-136a13f50c72" volumeName="kubernetes.io/configmap/7709b8a5-ca78-45d6-9c8c-136a13f50c72-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563363    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7ad82f0-86a1-43e3-8237-846a3aef0c88" volumeName="kubernetes.io/configmap/f7ad82f0-86a1-43e3-8237-846a3aef0c88-metrics-client-ca" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563379    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-kube-api-access-d57bx" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563426    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563442    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2343d79d-4146-4f1f-90be-ad84a256a4a5" volumeName="kubernetes.io/projected/2343d79d-4146-4f1f-90be-ad84a256a4a5-kube-api-access-tbhd4" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563455    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd722809-68b9-41b8-8a5e-814903a74299" volumeName="kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563466    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7ad82f0-86a1-43e3-8237-846a3aef0c88" volumeName="kubernetes.io/empty-dir/f7ad82f0-86a1-43e3-8237-846a3aef0c88-volume-directive-shadow" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563478    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563491    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2343d79d-4146-4f1f-90be-ad84a256a4a5" volumeName="kubernetes.io/empty-dir/2343d79d-4146-4f1f-90be-ad84a256a4a5-tmp-dir" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563506    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbb54d63-c3cc-4759-ba36-252e1ffc4d18" volumeName="kubernetes.io/configmap/bbb54d63-c3cc-4759-ba36-252e1ffc4d18-konnectivity-ca" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563517    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddd38ac5-1413-4345-9452-05936c2b31b8" volumeName="kubernetes.io/projected/ddd38ac5-1413-4345-9452-05936c2b31b8-kube-api-access-ndn6b" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563529    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7709b8a5-ca78-45d6-9c8c-136a13f50c72" volumeName="kubernetes.io/empty-dir/7709b8a5-ca78-45d6-9c8c-136a13f50c72-audit-log" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563541    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a195064-4c82-47ec-939e-8f0ac45fb0f6" volumeName="kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovnkube-config" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563554    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-grpc-tls" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563566    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563577    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30a15bca-3504-49fe-9ecf-c9860108905e" volumeName="kubernetes.io/secret/30a15bca-3504-49fe-9ecf-c9860108905e-monitoring-plugin-cert" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563589    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70b50531-ed83-4bac-bdf1-0992042e01a0" volumeName="kubernetes.io/configmap/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563602    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ecfafe4-94c0-451e-b646-54aa9dce7f46" volumeName="kubernetes.io/empty-dir/9ecfafe4-94c0-451e-b646-54aa9dce7f46-tmp" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563615    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ecfafe4-94c0-451e-b646-54aa9dce7f46" volumeName="kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-grpc-tls" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563627    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563638    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7709b8a5-ca78-45d6-9c8c-136a13f50c72" volumeName="kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-secret-metrics-server-tls" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563650    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b16518-f7fa-4ddf-ab14-ace75b8d127b" volumeName="kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-tls-assets" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563664    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="996125c2-4148-45f2-845d-33e743fdd73a" volumeName="kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563676    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="996125c2-4148-45f2-845d-33e743fdd73a" volumeName="kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-cni-binary-copy" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563688    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b" volumeName="kubernetes.io/projected/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-kube-api-access-255mf" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563700    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ecfafe4-94c0-451e-b646-54aa9dce7f46" volumeName="kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563712    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef" volumeName="kubernetes.io/secret/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563724    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" volumeName="kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563736    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b16518-f7fa-4ddf-ab14-ace75b8d127b" volumeName="kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-out" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563749    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b16518-f7fa-4ddf-ab14-ace75b8d127b" volumeName="kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-main-db" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563761    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b16518-f7fa-4ddf-ab14-ace75b8d127b" volumeName="kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563773    5837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c72d4793-688e-4c37-bb51-e0a9100a2203" volumeName="kubernetes.io/projected/c72d4793-688e-4c37-bb51-e0a9100a2203-kube-api-access-h77bx" seLinuxMountContext=""
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563785    5837 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 12:36:51.564919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.563794    5837 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 12:36:51.594357 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.594320    5837 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:36:51.594816 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.594728    5837 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 12:36:51.596994 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.596975    5837 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 12:36:51.597120 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.597002    5837 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 12:36:51.597120 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.597055    5837 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 12:36:51.597239 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:36:51.597112    5837 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 12:36:51.602837 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.602807    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 12:36:51.603522 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.603484    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 12:36:51.607466 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.607442    5837 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 12:36:51.613118 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.613084    5837 generic.go:356] "Generic (PLEG): container finished" podID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerID="efa532c5fe9cd30ec8a7deefcad58c09f85970b879b921fea96683d75c24e165" exitCode=0
Oct 30 12:36:51.613332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.613303    5837 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:36:51.618038 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.618010    5837 generic.go:356] "Generic (PLEG): container finished" podID="70b50531-ed83-4bac-bdf1-0992042e01a0" containerID="b62051c63b13e1269a481eb91c27900245f92a098f78ec6f01d687d908882914" exitCode=0
Oct 30 12:36:51.630145 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.630116    5837 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:36:51.630145 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.630122    5837 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:36:51.647475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.647440    5837 generic.go:356] "Generic (PLEG): container finished" podID="996125c2-4148-45f2-845d-33e743fdd73a" containerID="b92aa2f0a01ccf9b3e141af86eb31c0cab38c43aebaef73e0a617f7e07879497" exitCode=0
Oct 30 12:36:51.647475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.647468    5837 generic.go:356] "Generic (PLEG): container finished" podID="996125c2-4148-45f2-845d-33e743fdd73a" containerID="6c8a9fd4d969002b4cbf2e687d767e2a0fd1552a6e0f53436d78621ec0020648" exitCode=0
Oct 30 12:36:51.647475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.647478    5837 generic.go:356] "Generic (PLEG): container finished" podID="996125c2-4148-45f2-845d-33e743fdd73a" containerID="ba987709f8e53fb5a0674615363b6b01f666e57b6016c6cf7329c1faefd628da" exitCode=0
Oct 30 12:36:51.647761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.647489    5837 generic.go:356] "Generic (PLEG): container finished" podID="996125c2-4148-45f2-845d-33e743fdd73a" containerID="64d6b8248ca7de6cd08a8f6db1cd8cadc55bf2c3216bcfe5b538d4056b5832cc" exitCode=0
Oct 30 12:36:51.647761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.647499    5837 generic.go:356] "Generic (PLEG): container finished" podID="996125c2-4148-45f2-845d-33e743fdd73a" containerID="8d9e647ffdc3d281d2d984ba8247d017bdb03baa35ed34d350ba06caa671b763" exitCode=0
Oct 30 12:36:51.647761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.647508    5837 generic.go:356] "Generic (PLEG): container finished" podID="996125c2-4148-45f2-845d-33e743fdd73a" containerID="0eb65152424c6529d9bed8acaf3c86f17b04556d35778eb32484bba78ab21a86" exitCode=0
Oct 30 12:36:51.650877 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.650842    5837 generic.go:356] "Generic (PLEG): container finished" podID="6068dd25b70d52ed49d0937c3f3549a5" containerID="f15724b0f674456c003f00775aa7ed554b92566acd2a36acbf5989bcccdf14cc" exitCode=0
Oct 30 12:36:51.659543 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.659514    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 12:36:51.660173 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.660141    5837 generic.go:356] "Generic (PLEG): container finished" podID="0a195064-4c82-47ec-939e-8f0ac45fb0f6" containerID="07db58e1fe193b2ef936f10833c14323497196bde80555fa41f6e550f9e9a5c1" exitCode=1
Oct 30 12:36:51.667003 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.666980    5837 generic.go:356] "Generic (PLEG): container finished" podID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerID="d12dfc6d683790f1f80da3c4b6ef5105a72f9cea3026d150db56d8645f6d3206" exitCode=0
Oct 30 12:36:51.692816 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.692791    5837 manager.go:324] Recovery completed
Oct 30 12:36:51.697279 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:36:51.697255    5837 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 12:36:51.719050 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.719020    5837 cpu_manager.go:226] "Starting" policy="none"
Oct 30 12:36:51.719050 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.719043    5837 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 12:36:51.719277 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.719068    5837 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 12:36:51.719277 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.719234    5837 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 12:36:51.719277 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.719251    5837 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 12:36:51.719277 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.719278    5837 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 12:36:51.719503 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.719286    5837 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 12:36:51.719503 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.719293    5837 policy_none.go:50] "Start"
Oct 30 12:36:51.719503 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.719307    5837 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 12:36:51.719503 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.719320    5837 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 12:36:51.719681 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.719543    5837 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 12:36:51.719681 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.719556    5837 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 12:36:51.719681 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.719564    5837 policy_none.go:44] "Start"
Oct 30 12:36:51.726125 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.726091    5837 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 12:36:51.729936 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.729918    5837 manager.go:341] "Starting Device Plugin manager"
Oct 30 12:36:51.730604 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:36:51.729946    5837 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 12:36:51.730604 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.730603    5837 server.go:87] "Starting device plugin registration server"
Oct 30 12:36:51.730891 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.730873    5837 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 12:36:51.730937 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.730885    5837 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 12:36:51.731020 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.731005    5837 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 12:36:51.731117 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.731093    5837 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 12:36:51.731190 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.731129    5837 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 12:36:51.731190 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.731134    5837 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 12:36:51.731283 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.731183    5837 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:36:51.73111537Z","UUID":"2b00acc1-1bcb-4378-a58f-7d67c45b9a06","Handler":null,"Name":"","Endpoint":""}
Oct 30 12:36:51.732733 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.732707    5837 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 12:36:51.732829 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.732747    5837 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 12:36:51.733198 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:36:51.733165    5837 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 12:36:51.831525 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.831407    5837 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:36:51.833472 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.833450    5837 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-228.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:36:51.833609 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.833484    5837 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-228.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:36:51.833609 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.833502    5837 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-228.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:36:51.833609 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.833563    5837 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-228.ec2.internal"
Oct 30 12:36:51.873968 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.873937    5837 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-228.ec2.internal"
Oct 30 12:36:51.874110 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.874020    5837 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-228.ec2.internal"
Oct 30 12:36:51.897907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.897852    5837 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal"]
Oct 30 12:36:51.898558 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.898542    5837 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:51.898558 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.898538    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal" event={"ID":"72d3c22f1a6da7d8f3667e690d2cfffa","Type":"ContainerStarted","Data":"9bbf4a84f3c357de558a421f4a14ac46018f64120049285e6dbfb05de1aeea0d"}
Oct 30 12:36:51.898657 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.898575    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal" event={"ID":"72d3c22f1a6da7d8f3667e690d2cfffa","Type":"ContainerStarted","Data":"0e3f71bc99acfb4c14662813364519fafe4f46385b1fbf306688db96514f26ca"}
Oct 30 12:36:51.898657 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.898588    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal" event={"ID":"6068dd25b70d52ed49d0937c3f3549a5","Type":"ContainerStarted","Data":"9d6a5111481351fa18152f5aa1b847e1f9456f2cb9f91e8bd911ed8f5dce39d7"}
Oct 30 12:36:51.898657 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.898601    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal" event={"ID":"6068dd25b70d52ed49d0937c3f3549a5","Type":"ContainerDied","Data":"f15724b0f674456c003f00775aa7ed554b92566acd2a36acbf5989bcccdf14cc"}
Oct 30 12:36:51.898657 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.898613    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal" event={"ID":"6068dd25b70d52ed49d0937c3f3549a5","Type":"ContainerStarted","Data":"560843455db2355b850de37a5d81ae51e5429266263244dfb720309e6095a59c"}
Oct 30 12:36:51.898793 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.898761    5837 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:51.926468 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.926441    5837 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:36:51.926617 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:36:51.926497    5837 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-228.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:51.934249 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:51.934230    5837 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:36:51.934366 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:36:51.934273    5837 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:52.009623 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.009593    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6068dd25b70d52ed49d0937c3f3549a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal\" (UID: \"6068dd25b70d52ed49d0937c3f3549a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:52.009623 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.009628    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6068dd25b70d52ed49d0937c3f3549a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal\" (UID: \"6068dd25b70d52ed49d0937c3f3549a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:52.009814 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.009645    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72d3c22f1a6da7d8f3667e690d2cfffa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-228.ec2.internal\" (UID: \"72d3c22f1a6da7d8f3667e690d2cfffa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:52.110309 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.110222    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6068dd25b70d52ed49d0937c3f3549a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal\" (UID: \"6068dd25b70d52ed49d0937c3f3549a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:52.110309 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.110275    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6068dd25b70d52ed49d0937c3f3549a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal\" (UID: \"6068dd25b70d52ed49d0937c3f3549a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:52.110309 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.110294    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72d3c22f1a6da7d8f3667e690d2cfffa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-228.ec2.internal\" (UID: \"72d3c22f1a6da7d8f3667e690d2cfffa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:52.110576 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.110230    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6068dd25b70d52ed49d0937c3f3549a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal\" (UID: \"6068dd25b70d52ed49d0937c3f3549a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:52.110576 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.110352    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6068dd25b70d52ed49d0937c3f3549a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal\" (UID: \"6068dd25b70d52ed49d0937c3f3549a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:52.110576 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.110332    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72d3c22f1a6da7d8f3667e690d2cfffa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-228.ec2.internal\" (UID: \"72d3c22f1a6da7d8f3667e690d2cfffa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:52.543202 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.543151    5837 apiserver.go:52] "Watching apiserver"
Oct 30 12:36:52.593876 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.593843    5837 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 12:36:52.594588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.594563    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zh8sx","openshift-ingress-canary/ingress-canary-qmqdw","openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w","openshift-monitoring/metrics-server-6f9858f786-6dftr","kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal","openshift-cluster-node-tuning-operator/tuned-jsgw4","openshift-dns/node-resolver-84tjf","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v","openshift-image-registry/node-ca-p9gs7","openshift-monitoring/alertmanager-main-0","openshift-multus/network-metrics-daemon-v4ttl","openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs","openshift-multus/multus-5dfvg","openshift-network-diagnostics/network-check-target-chqf2","openshift-ovn-kubernetes/ovnkube-node-9trzv","openshift-insights/insights-runtime-extractor-dg6tk","openshift-monitoring/node-exporter-9gvsj","openshift-multus/multus-additional-cni-plugins-x6hvr","openshift-network-operator/iptables-alerter-vmqth","openshift-monitoring/thanos-querier-56878775cd-4pz7n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal","kube-system/global-pull-secret-syncer-kd88h","kube-system/konnectivity-agent-n6d9z"]
Oct 30 12:36:52.611480 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.611444    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 12:36:52.611480 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.611459    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 12:36:52.611480 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.611459    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 12:36:52.611480 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.611443    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:36:52.611731 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.611492    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 12:36:52.611731 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.611504    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 12:36:52.611731 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.611491    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 12:36:52.611731 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.611512    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 12:36:52.611731 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.611537    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 12:36:52.620550 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.620523    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 12:36:52.620662 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.620558    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 12:36:52.620662 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.620645    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 12:36:52.622871 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.622857    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 12:36:52.623023 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.623011    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:36:52.623097 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.623084    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 12:36:52.623154 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.623143    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 12:36:52.623281 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.623269    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 12:36:52.623371 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.623360    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 12:36:52.627110 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.627090    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 12:36:52.627308 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.627294    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 12:36:52.627379 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.627326    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 12:36:52.627379 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.627359    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 12:36:52.627652 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.627639    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 12:36:52.627716 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.627697    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:36:52.627768 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.627756    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 12:36:52.627812 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.627756    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:36:52.627860 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.627848    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 12:36:52.627903 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.627881    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 12:36:52.628000 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.627984    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 12:36:52.628057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.628027    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 12:36:52.628057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.628050    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 12:36:52.628150 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.628081    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 12:36:52.628150 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.628094    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 12:36:52.628150 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.628113    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 12:36:52.628288 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.628173    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 12:36:52.628338 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.628310    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 12:36:52.628338 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.628313    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 12:36:52.628453 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.628357    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 12:36:52.628453 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.628357    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 12:36:52.628648 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.628630    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 12:36:52.628648 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.628639    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 12:36:52.629020 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.629007    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 12:36:52.640997 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.640973    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 12:36:52.641134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.641003    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 12:36:52.641184 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.641165    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 12:36:52.662693 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.662668    5837 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 12:36:52.673512 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.673485    5837 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:52.673631 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.673598    5837 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:52.693870 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.693838    5837 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:36:52.694013 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:36:52.693902    5837 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-228.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:52.694321 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.694304    5837 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:36:52.694444 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:36:52.694359    5837 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-228.ec2.internal"
Oct 30 12:36:52.704508 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.704476    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 12:36:52.704613 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.704590    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 12:36:52.704699 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.704654    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 12:36:52.704774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.704689    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 12:36:52.704846 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.704827    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 12:36:52.704909 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.704855    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 12:36:52.704997 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.704982    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 12:36:52.705034 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.704970    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 12:36:52.705151 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705136    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 12:36:52.705223 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705138    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 12:36:52.705291 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705280    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2vwzh\""
Oct 30 12:36:52.705345 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705332    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 12:36:52.705466 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705448    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 12:36:52.705509 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705464    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bg2k7\""
Oct 30 12:36:52.705565 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705554    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 12:36:52.705613 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705566    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 12:36:52.705613 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705592    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 12:36:52.705715 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705676    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t86w7\""
Oct 30 12:36:52.705715 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705698    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-562t6\""
Oct 30 12:36:52.705715 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705683    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 12:36:52.705814 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705728    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-whrjh\""
Oct 30 12:36:52.705814 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705750    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 12:36:52.705814 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705773    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 12:36:52.705937 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705815    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 12:36:52.705937 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.705873    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 12:36:52.706179 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706167    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 12:36:52.706328 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706312    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 12:36:52.706378 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706332    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d504fh0uqulkd\""
Oct 30 12:36:52.706378 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706352    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 12:36:52.706378 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706356    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 12:36:52.706378 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706353    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 12:36:52.706596 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706456    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 12:36:52.706596 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706465    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-scxms\""
Oct 30 12:36:52.706596 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706474    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-t6s7j\""
Oct 30 12:36:52.706596 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706497    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qjz2q\""
Oct 30 12:36:52.706596 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706524    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 12:36:52.706770 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706649    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-s26wd\""
Oct 30 12:36:52.706770 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706672    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 12:36:52.706891 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706868    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 12:36:52.707021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706937    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 12:36:52.707021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706939    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x6m2k\""
Oct 30 12:36:52.707021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706939    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 12:36:52.707021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706976    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 12:36:52.707021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706984    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6qrlt\""
Oct 30 12:36:52.707021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706993    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 12:36:52.707021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707014    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tz7hb\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.706987    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707063    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rkrpr\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707064    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-q874n\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707071    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707035    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dtxmh\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707075    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707098    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-54vc4\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707100    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gmdlh\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707143    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kkflg\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707019    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-w8qvh\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707026    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707154    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-87th4\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707065    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wbnrm\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707123    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6dg1ritap5cqn\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707146    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707108    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1rld844cc7msd\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707164    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zf26c\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707217    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 12:36:52.707318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.707113    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pdckv\""
Oct 30 12:36:52.713195 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713176    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-kubelet\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.713304 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713204    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-secret-metrics-server-tls\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:52.713304 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713221    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.713304 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713234    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-host\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.713304 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713253    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:52.713524 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713506    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-cnibin\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.713592 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713547    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.713645 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713596    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-secret-metrics-server-tls\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:52.713645 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713610    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:52.713645 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713618    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.713778 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713667    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/233c3bd3-01bc-4c71-8bc4-611933e34539-iptables-alerter-script\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:36:52.713778 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713700    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-etc-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.713778 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713728    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-cnibin\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.713919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713805    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.713919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713822    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.713919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713842    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-tls\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.713919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713874    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.713919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713875    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/233c3bd3-01bc-4c71-8bc4-611933e34539-iptables-alerter-script\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:36:52.714106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713949    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-wtmp\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.714106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713968    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-cni-netd\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.714106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.713987    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbhd4\" (UniqueName: \"kubernetes.io/projected/2343d79d-4146-4f1f-90be-ad84a256a4a5-kube-api-access-tbhd4\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:52.714106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714007    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7709b8a5-ca78-45d6-9c8c-136a13f50c72-metrics-server-audit-profiles\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:52.714106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714015    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-tls\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.714106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714042    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhm4h\" (UniqueName: \"kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h\") pod \"network-check-target-chqf2\" (UID: \"cd722809-68b9-41b8-8a5e-814903a74299\") " pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:52.714375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714156    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-root\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.714375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714177    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.714375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714283    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7709b8a5-ca78-45d6-9c8c-136a13f50c72-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:52.714375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714294    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7709b8a5-ca78-45d6-9c8c-136a13f50c72-metrics-server-audit-profiles\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:52.714539 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714370    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-registration-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.714539 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714440    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ddd38ac5-1413-4345-9452-05936c2b31b8-cert\") pod \"ingress-canary-qmqdw\" (UID: \"ddd38ac5-1413-4345-9452-05936c2b31b8\") " pod="openshift-ingress-canary/ingress-canary-qmqdw"
Oct 30 12:36:52.714539 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714473    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.714539 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714481    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2343d79d-4146-4f1f-90be-ad84a256a4a5-metrics-tls\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:52.714539 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714513    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7709b8a5-ca78-45d6-9c8c-136a13f50c72-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:52.714539 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714526    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2343d79d-4146-4f1f-90be-ad84a256a4a5-tmp-dir\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:52.714753 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714560    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.714753 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714587    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-kubernetes\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.714753 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714598    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2343d79d-4146-4f1f-90be-ad84a256a4a5-tmp-dir\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:52.714753 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714606    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-hostroot\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.714753 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714624    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.714753 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714673    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ddd38ac5-1413-4345-9452-05936c2b31b8-cert\") pod \"ingress-canary-qmqdw\" (UID: \"ddd38ac5-1413-4345-9452-05936c2b31b8\") " pod="openshift-ingress-canary/ingress-canary-qmqdw"
Oct 30 12:36:52.714753 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714678    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2343d79d-4146-4f1f-90be-ad84a256a4a5-metrics-tls\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:52.714753 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714733    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714765    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-run\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714775    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714793    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndn6b\" (UniqueName: \"kubernetes.io/projected/ddd38ac5-1413-4345-9452-05936c2b31b8-kube-api-access-ndn6b\") pod \"ingress-canary-qmqdw\" (UID: \"ddd38ac5-1413-4345-9452-05936c2b31b8\") " pod="openshift-ingress-canary/ingress-canary-qmqdw"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714804    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714825    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714823    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8w42\" (UniqueName: \"kubernetes.io/projected/9ecfafe4-94c0-451e-b646-54aa9dce7f46-kube-api-access-z8w42\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714852    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-ovn\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714868    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714883    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714913    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.714992    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715021    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v766r\" (UniqueName: \"kubernetes.io/projected/233c3bd3-01bc-4c71-8bc4-611933e34539-kube-api-access-v766r\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715031    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715048    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-kubelet\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715071    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-systemd-units\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715075    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715091    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715125    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715111    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqncb\" (UniqueName: \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-kube-api-access-vqncb\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715171    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-device-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715222    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715263    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715282    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-volume\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715298    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-sys-fs\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715312    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysconfig\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715327    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c24sf\" (UniqueName: \"kubernetes.io/projected/eb31cffb-32c0-4d28-9947-325621908909-kube-api-access-c24sf\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715369    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtx6n\" (UniqueName: \"kubernetes.io/projected/996125c2-4148-45f2-845d-33e743fdd73a-kube-api-access-xtx6n\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715449    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715467    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.715465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715483    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/233c3bd3-01bc-4c71-8bc4-611933e34539-host-slash\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715508    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-slash\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715522    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-volume\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715533    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovnkube-script-lib\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715556    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715573    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715604    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpt8\" (UniqueName: \"kubernetes.io/projected/30a15bca-3504-49fe-9ecf-c9860108905e-kube-api-access-mtpt8\") pod \"monitoring-plugin-77f4f76594-wqwgs\" (UID: \"30a15bca-3504-49fe-9ecf-c9860108905e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715771    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715807    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2343d79d-4146-4f1f-90be-ad84a256a4a5-config-volume\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715835    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715876    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-client-ca-bundle\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715922    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.715954    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovnkube-script-lib\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716018    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-dbus\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716020    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2343d79d-4146-4f1f-90be-ad84a256a4a5-config-volume\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716060    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-client-ca-bundle\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716084    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkqw9\" (UniqueName: \"kubernetes.io/projected/e383bb23-4967-47e8-afdf-937f840aea60-kube-api-access-nkqw9\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716103    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716106    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-grpc-tls\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716137    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-socket-dir-parent\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716165    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-k8s-cni-cncf-io\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716201    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716209    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-cni-multus\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716270    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-var-lib-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716318    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-grpc-tls\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716342    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-log-socket\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716366    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716382    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-multus-certs\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716417    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-lib-modules\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716453    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhl5\" (UniqueName: \"kubernetes.io/projected/e4fe0229-c475-400a-b6c0-48d80476b3b5-kube-api-access-rfhl5\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716478    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.716531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716500    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c72d4793-688e-4c37-bb51-e0a9100a2203-host\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716659    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-systemd\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716663    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716689    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-secret-metrics-server-client-certs\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716740    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7ad82f0-86a1-43e3-8237-846a3aef0c88-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716833    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716858    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716874    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7709b8a5-ca78-45d6-9c8c-136a13f50c72-secret-metrics-server-client-certs\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716875    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716901    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-web-config\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716917    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7709b8a5-ca78-45d6-9c8c-136a13f50c72-audit-log\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716937    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc6np\" (UniqueName: \"kubernetes.io/projected/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-api-access-gc6np\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716953    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70b50531-ed83-4bac-bdf1-0992042e01a0-metrics-client-ca\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716951    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.716962    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7ad82f0-86a1-43e3-8237-846a3aef0c88-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717006    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7709b8a5-ca78-45d6-9c8c-136a13f50c72-audit-log\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717042    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-etc-kubernetes\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717056    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717094    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70b50531-ed83-4bac-bdf1-0992042e01a0-metrics-client-ca\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717071    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717129    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-var-lib-kubelet\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717147    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717159    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-web-config\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717160    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717163    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c72d4793-688e-4c37-bb51-e0a9100a2203-serviceca\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717202    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-run-netns\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717230    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovn-node-metrics-cert\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717256    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-netns\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717280    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717287    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717314    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysctl-conf\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717344    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-systemd\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717431    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717459    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717477    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecfafe4-94c0-451e-b646-54aa9dce7f46-tmp\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717473    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717516    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-255mf\" (UniqueName: \"kubernetes.io/projected/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-kube-api-access-255mf\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717515    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovn-node-metrics-cert\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717516    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.717515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717534    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f7ad82f0-86a1-43e3-8237-846a3aef0c88-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717574    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecfafe4-94c0-451e-b646-54aa9dce7f46-tmp\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717580    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717611    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-cni-bin\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717618    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717670    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-cni-bin\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717690    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74n4f\" (UniqueName: \"kubernetes.io/projected/ca860770-e681-4dae-806f-ef6de17bed8e-kube-api-access-74n4f\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717758    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f7ad82f0-86a1-43e3-8237-846a3aef0c88-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717802    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bbb54d63-c3cc-4759-ba36-252e1ffc4d18-agent-certs\") pod \"konnectivity-agent-n6d9z\" (UID: \"bbb54d63-c3cc-4759-ba36-252e1ffc4d18\") " pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717830    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717831    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717918    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8gn6\" (UniqueName: \"kubernetes.io/projected/0a195064-4c82-47ec-939e-8f0ac45fb0f6-kube-api-access-h8gn6\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717920    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-original-pull-secret\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.717987    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c72d4793-688e-4c37-bb51-e0a9100a2203-serviceca\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718019    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14b22d89-19e4-4dbb-a031-91b5fe337785-cni-binary-copy\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718024    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bbb54d63-c3cc-4759-ba36-252e1ffc4d18-agent-certs\") pod \"konnectivity-agent-n6d9z\" (UID: \"bbb54d63-c3cc-4759-ba36-252e1ffc4d18\") " pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718102    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718112    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-metrics-certs\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718178    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718233    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14b22d89-19e4-4dbb-a031-91b5fe337785-cni-binary-copy\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718269    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-kubelet-config\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718297    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e383bb23-4967-47e8-afdf-937f840aea60-crio-socket\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718300    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718326    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ecfafe4-94c0-451e-b646-54aa9dce7f46-metrics-client-ca\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718354    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718385    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718424    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30a15bca-3504-49fe-9ecf-c9860108905e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wqwgs\" (UID: \"30a15bca-3504-49fe-9ecf-c9860108905e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718454    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-socket-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718489    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-textfile\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718518    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhpx\" (UniqueName: \"kubernetes.io/projected/70b50531-ed83-4bac-bdf1-0992042e01a0-kube-api-access-xqhpx\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718540    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ecfafe4-94c0-451e-b646-54aa9dce7f46-metrics-client-ca\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718545    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e383bb23-4967-47e8-afdf-937f840aea60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718549    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718573    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718596    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718594    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30a15bca-3504-49fe-9ecf-c9860108905e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wqwgs\" (UID: \"30a15bca-3504-49fe-9ecf-c9860108905e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718613    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718683    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-textfile\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718762    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e383bb23-4967-47e8-afdf-937f840aea60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718767    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysctl-d\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718776    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718781    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718779    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718809    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-sys\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718833    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e4fe0229-c475-400a-b6c0-48d80476b3b5-hosts-file\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718850    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718880    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-daemon-config\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718907    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbg47\" (UniqueName: \"kubernetes.io/projected/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-kube-api-access-lbg47\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718933    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718963    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.718959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.718990    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-774wq\" (UniqueName: \"kubernetes.io/projected/14b22d89-19e4-4dbb-a031-91b5fe337785-kube-api-access-774wq\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719005    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-daemon-config\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719048    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719083    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719110    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-cni-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719157    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719184    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719207    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719256    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719266    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719431    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719450    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719500    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bbb54d63-c3cc-4759-ba36-252e1ffc4d18-konnectivity-ca\") pod \"konnectivity-agent-n6d9z\" (UID: \"bbb54d63-c3cc-4759-ba36-252e1ffc4d18\") " pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719730    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bbb54d63-c3cc-4759-ba36-252e1ffc4d18-konnectivity-ca\") pod \"konnectivity-agent-n6d9z\" (UID: \"bbb54d63-c3cc-4759-ba36-252e1ffc4d18\") " pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719778    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e383bb23-4967-47e8-afdf-937f840aea60-data-volume\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719808    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719836    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719868    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-tls\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719923    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-env-overrides\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719872    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e383bb23-4967-47e8-afdf-937f840aea60-data-volume\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.719947    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-system-cni-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720017    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720022    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720026    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9ecfafe4-94c0-451e-b646-54aa9dce7f46-secret-thanos-querier-tls\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720064    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-sys\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720068    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720178    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-env-overrides\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720181    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720259    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720296    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovnkube-config\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720339    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-out\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720365    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720427    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb31cffb-32c0-4d28-9947-325621908909-etc-tuned\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720436    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-out\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720478    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-node-log\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720498    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb31cffb-32c0-4d28-9947-325621908909-etc-tuned\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720538    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-conf-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720549    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a195064-4c82-47ec-939e-8f0ac45fb0f6-ovnkube-config\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720566    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720606    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-modprobe-d\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720639    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb31cffb-32c0-4d28-9947-325621908909-tmp\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720695    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4fe0229-c475-400a-b6c0-48d80476b3b5-tmp-dir\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720729    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb31cffb-32c0-4d28-9947-325621908909-tmp\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720784    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-system-cni-dir\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720795    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4fe0229-c475-400a-b6c0-48d80476b3b5-tmp-dir\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720822    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720926    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l6nw\" (UniqueName: \"kubernetes.io/projected/7709b8a5-ca78-45d6-9c8c-136a13f50c72-kube-api-access-8l6nw\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720945    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720962    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d57bx\" (UniqueName: \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-kube-api-access-d57bx\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.720985    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h77bx\" (UniqueName: \"kubernetes.io/projected/c72d4793-688e-4c37-bb51-e0a9100a2203-kube-api-access-h77bx\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:36:52.721014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.721005    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/996125c2-4148-45f2-845d-33e743fdd73a-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.722482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.721095    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e383bb23-4967-47e8-afdf-937f840aea60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:52.722482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.721174    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:52.722482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.721249    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e383bb23-4967-47e8-afdf-937f840aea60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:52.722922 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.722903    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.736996 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.736966    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbhd4\" (UniqueName: \"kubernetes.io/projected/2343d79d-4146-4f1f-90be-ad84a256a4a5-kube-api-access-tbhd4\") pod \"dns-default-zh8sx\" (UID: \"2343d79d-4146-4f1f-90be-ad84a256a4a5\") " pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:52.737099 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.737079    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-255mf\" (UniqueName: \"kubernetes.io/projected/9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b-kube-api-access-255mf\") pod \"network-metrics-daemon-v4ttl\" (UID: \"9a7d1b97-ad0b-4e7f-b2e2-b82bbd76fe8b\") " pod="openshift-multus/network-metrics-daemon-v4ttl"
Oct 30 12:36:52.739380 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.739355    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfhl5\" (UniqueName: \"kubernetes.io/projected/e4fe0229-c475-400a-b6c0-48d80476b3b5-kube-api-access-rfhl5\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:36:52.739519 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.739357    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndn6b\" (UniqueName: \"kubernetes.io/projected/ddd38ac5-1413-4345-9452-05936c2b31b8-kube-api-access-ndn6b\") pod \"ingress-canary-qmqdw\" (UID: \"ddd38ac5-1413-4345-9452-05936c2b31b8\") " pod="openshift-ingress-canary/ingress-canary-qmqdw"
Oct 30 12:36:52.739519 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.739500    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v766r\" (UniqueName: \"kubernetes.io/projected/233c3bd3-01bc-4c71-8bc4-611933e34539-kube-api-access-v766r\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:36:52.739626 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.739546    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c24sf\" (UniqueName: \"kubernetes.io/projected/eb31cffb-32c0-4d28-9947-325621908909-kube-api-access-c24sf\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.739692 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.739637    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhpx\" (UniqueName: \"kubernetes.io/projected/70b50531-ed83-4bac-bdf1-0992042e01a0-kube-api-access-xqhpx\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.739803 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.739737    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpt8\" (UniqueName: \"kubernetes.io/projected/30a15bca-3504-49fe-9ecf-c9860108905e-kube-api-access-mtpt8\") pod \"monitoring-plugin-77f4f76594-wqwgs\" (UID: \"30a15bca-3504-49fe-9ecf-c9860108905e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"
Oct 30 12:36:52.740290 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.740275    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8w42\" (UniqueName: \"kubernetes.io/projected/9ecfafe4-94c0-451e-b646-54aa9dce7f46-kube-api-access-z8w42\") pod \"thanos-querier-56878775cd-4pz7n\" (UID: \"9ecfafe4-94c0-451e-b646-54aa9dce7f46\") " pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:52.741290 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.741252    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8gn6\" (UniqueName: \"kubernetes.io/projected/0a195064-4c82-47ec-939e-8f0ac45fb0f6-kube-api-access-h8gn6\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.741416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.741346    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbg47\" (UniqueName: \"kubernetes.io/projected/a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef-kube-api-access-lbg47\") pod \"openshift-state-metrics-7bd7489957-ql52w\" (UID: \"a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ql52w"
Oct 30 12:36:52.741480 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.741418    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-774wq\" (UniqueName: \"kubernetes.io/projected/14b22d89-19e4-4dbb-a031-91b5fe337785-kube-api-access-774wq\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.741537 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.741473    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtx6n\" (UniqueName: \"kubernetes.io/projected/996125c2-4148-45f2-845d-33e743fdd73a-kube-api-access-xtx6n\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.741678 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.741661    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l6nw\" (UniqueName: \"kubernetes.io/projected/7709b8a5-ca78-45d6-9c8c-136a13f50c72-kube-api-access-8l6nw\") pod \"metrics-server-6f9858f786-6dftr\" (UID: \"7709b8a5-ca78-45d6-9c8c-136a13f50c72\") " pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:52.741749 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.741659    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhm4h\" (UniqueName: \"kubernetes.io/projected/cd722809-68b9-41b8-8a5e-814903a74299-kube-api-access-fhm4h\") pod \"network-check-target-chqf2\" (UID: \"cd722809-68b9-41b8-8a5e-814903a74299\") " pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:36:52.741802 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.741786    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74n4f\" (UniqueName: \"kubernetes.io/projected/ca860770-e681-4dae-806f-ef6de17bed8e-kube-api-access-74n4f\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.741878 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.741857    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkqw9\" (UniqueName: \"kubernetes.io/projected/e383bb23-4967-47e8-afdf-937f840aea60-kube-api-access-nkqw9\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:52.742228 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.742213    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc6np\" (UniqueName: \"kubernetes.io/projected/f7ad82f0-86a1-43e3-8237-846a3aef0c88-kube-api-access-gc6np\") pod \"kube-state-metrics-5dc8d7d89b-gb5g9\" (UID: \"f7ad82f0-86a1-43e3-8237-846a3aef0c88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gb5g9"
Oct 30 12:36:52.742967 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.742945    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h77bx\" (UniqueName: \"kubernetes.io/projected/c72d4793-688e-4c37-bb51-e0a9100a2203-kube-api-access-h77bx\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:36:52.743245 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.743229    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d57bx\" (UniqueName: \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-kube-api-access-d57bx\") pod \"prometheus-k8s-0\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:52.743379 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.743362    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqncb\" (UniqueName: \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-kube-api-access-vqncb\") pod \"alertmanager-main-0\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:36:52.822026 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.821939    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-cni-bin\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.822026 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.821975    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-cni-bin\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.822196 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822017    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-kubelet-config\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:52.822196 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822063    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-cni-bin\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.822196 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822076    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e383bb23-4967-47e8-afdf-937f840aea60-crio-socket\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:52.822196 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822105    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-kubelet-config\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:52.822196 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822117    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-socket-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.822196 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822145    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysctl-d\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.822196 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822168    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-sys\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.822196 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822192    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e4fe0229-c475-400a-b6c0-48d80476b3b5-hosts-file\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:36:52.822196 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822168    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-socket-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822205    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e383bb23-4967-47e8-afdf-937f840aea60-crio-socket\") pod \"insights-runtime-extractor-dg6tk\" (UID: \"e383bb23-4967-47e8-afdf-937f840aea60\") " pod="openshift-insights/insights-runtime-extractor-dg6tk"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822190    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-cni-bin\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822257    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822287    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e4fe0229-c475-400a-b6c0-48d80476b3b5-hosts-file\") pod \"node-resolver-84tjf\" (UID: \"e4fe0229-c475-400a-b6c0-48d80476b3b5\") " pod="openshift-dns/node-resolver-84tjf"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822308    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-cni-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822313    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysctl-d\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822264    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-sys\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822292    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822352    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-system-cni-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822352    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-cni-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822382    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-sys\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822447    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-system-cni-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822466    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-sys\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822499    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-node-log\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822530    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-conf-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822544    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822572    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-modprobe-d\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822585    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-node-log\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822590    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-conf-dir\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.822597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822597    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-system-cni-dir\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822636    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822644    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-kubelet\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822662    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-modprobe-d\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822686    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-system-cni-dir\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822690    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822692    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-kubelet\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822718    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-host\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822744    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-cnibin\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822770    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-etc-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822797    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-host\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822799    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-cnibin\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822815    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-cnibin\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822829    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822845    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-cnibin\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822850    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-wtmp\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822797    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822876    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/996125c2-4148-45f2-845d-33e743fdd73a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6hvr\" (UID: \"996125c2-4148-45f2-845d-33e743fdd73a\") " pod="openshift-multus/multus-additional-cni-plugins-x6hvr"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822879    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-cni-netd\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822938    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-root\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822944    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-node-exporter-wtmp\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822912    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-cni-netd\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822960    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-etc-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822984    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70b50531-ed83-4bac-bdf1-0992042e01a0-root\") pod \"node-exporter-9gvsj\" (UID: \"70b50531-ed83-4bac-bdf1-0992042e01a0\") " pod="openshift-monitoring/node-exporter-9gvsj"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.822965    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-registration-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823065    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-kubernetes\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823081    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-hostroot\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823083    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-registration-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823100    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-run\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823135    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-kubernetes\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823164    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-hostroot\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823176    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-ovn\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823194    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-run\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823217    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-kubelet\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823241    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-systemd-units\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823241    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-ovn\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823282    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823322    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-device-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823317    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-kubelet\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823342    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-sys-fs\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823372    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823440    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysconfig\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823450    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-sys-fs\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823480    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca860770-e681-4dae-806f-ef6de17bed8e-device-dir\") pod \"aws-ebs-csi-driver-node-mn55v\" (UID: \"ca860770-e681-4dae-806f-ef6de17bed8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn55v"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823486    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-systemd-units\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823372    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysconfig\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823543    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/233c3bd3-01bc-4c71-8bc4-611933e34539-host-slash\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823564    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-slash\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.823559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823598    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-dbus\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823633    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-socket-dir-parent\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823650    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-k8s-cni-cncf-io\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823659    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-slash\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823672    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-cni-multus\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823697    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-var-lib-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823702    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-var-lib-cni-multus\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823627    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/233c3bd3-01bc-4c71-8bc4-611933e34539-host-slash\") pod \"iptables-alerter-vmqth\" (UID: \"233c3bd3-01bc-4c71-8bc4-611933e34539\") " pod="openshift-network-operator/iptables-alerter-vmqth"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823739    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-k8s-cni-cncf-io\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823763    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4-dbus\") pod \"global-pull-secret-syncer-kd88h\" (UID: \"0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4\") " pod="kube-system/global-pull-secret-syncer-kd88h"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823759    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-multus-socket-dir-parent\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823784    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-var-lib-openvswitch\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823822    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-log-socket\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823847    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823867    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-multus-certs\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823877    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-log-socket\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823886    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-lib-modules\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823897    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823921    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c72d4793-688e-4c37-bb51-e0a9100a2203-host\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823911    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-multus-certs\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823951    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-systemd\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823975    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-lib-modules\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.823983    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-etc-kubernetes\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.824003    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c72d4793-688e-4c37-bb51-e0a9100a2203-host\") pod \"node-ca-p9gs7\" (UID: \"c72d4793-688e-4c37-bb51-e0a9100a2203\") " pod="openshift-image-registry/node-ca-p9gs7"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.824009    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-var-lib-kubelet\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.824009    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-run-systemd\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.824020    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-etc-kubernetes\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.824048    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-run-netns\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.824061    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-var-lib-kubelet\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.824064    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-netns\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.824081    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14b22d89-19e4-4dbb-a031-91b5fe337785-host-run-netns\") pod \"multus-5dfvg\" (UID: \"14b22d89-19e4-4dbb-a031-91b5fe337785\") " pod="openshift-multus/multus-5dfvg"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.824096    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysctl-conf\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.824098    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a195064-4c82-47ec-939e-8f0ac45fb0f6-host-run-netns\") pod \"ovnkube-node-9trzv\" (UID: \"0a195064-4c82-47ec-939e-8f0ac45fb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.824121    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-systemd\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.824164    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-systemd\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:52.825116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:52.824227    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb31cffb-32c0-4d28-9947-325621908909-etc-sysctl-conf\") pod \"tuned-jsgw4\" (UID: \"eb31cffb-32c0-4d28-9947-325621908909\") " pod="openshift-cluster-node-tuning-operator/tuned-jsgw4"
Oct 30 12:36:53.745722 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:53.745683    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqwgs"
Oct 30 12:36:54.195215 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:54.195134    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-56878775cd-4pz7n"
Oct 30 12:36:54.578031 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:54.577994    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zh8sx"
Oct 30 12:36:56.469208 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:56.469177    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:36:57.033277 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:57.033237    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6f9858f786-6dftr"
Oct 30 12:36:58.652050 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:58.652021    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:59.612081 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:59.612051    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:36:59.612329 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:36:59.612296    5837 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:37:00.590412 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:00.590360    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:37:03.409821 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:03.409742    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qmqdw_ddd38ac5-1413-4345-9452-05936c2b31b8/serve-healthcheck-canary/0.log"
Oct 30 12:37:05.233885 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.233846    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:37:05.234505 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.234468    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="alertmanager" containerID="cri-o://c00678908b8509631d5735197a35b3c60d7c5b2e9b6e1a1da46fc5043089fcc0" gracePeriod=120
Oct 30 12:37:05.234505 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.234487    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="kube-rbac-proxy" containerID="cri-o://7d6311147738646a2b8099040a3efa4c265f0939d1d144f571e3ad1df8c54953" gracePeriod=120
Oct 30 12:37:05.234739 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.234533    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="config-reloader" containerID="cri-o://c101f9bdf653ede0666117b92796b7a8f3855231edb0ac2305d73f3b5e528e36" gracePeriod=120
Oct 30 12:37:05.234739 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.234533    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="kube-rbac-proxy-web" containerID="cri-o://a22b4e94bbcd95169aed0f2e64ee27c20b883de077763946b807e9840fca4732" gracePeriod=120
Oct 30 12:37:05.234739 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.234479    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="prom-label-proxy" containerID="cri-o://ba0cde52853918504e8e2b02c015fdf789a1964c1f93caa408dd1eeb878803ad" gracePeriod=120
Oct 30 12:37:05.234739 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.234512    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="kube-rbac-proxy-metric" containerID="cri-o://01cb6c1f41abb8ee8d23626a841926d92a98a3abfa4fc270a32683230d6eda2f" gracePeriod=120
Oct 30 12:37:05.728834 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.728752    5837 generic.go:356] "Generic (PLEG): container finished" podID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerID="ba0cde52853918504e8e2b02c015fdf789a1964c1f93caa408dd1eeb878803ad" exitCode=0
Oct 30 12:37:05.728834 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.728776    5837 generic.go:356] "Generic (PLEG): container finished" podID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerID="01cb6c1f41abb8ee8d23626a841926d92a98a3abfa4fc270a32683230d6eda2f" exitCode=0
Oct 30 12:37:05.728834 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.728782    5837 generic.go:356] "Generic (PLEG): container finished" podID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerID="7d6311147738646a2b8099040a3efa4c265f0939d1d144f571e3ad1df8c54953" exitCode=0
Oct 30 12:37:05.728834 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.728788    5837 generic.go:356] "Generic (PLEG): container finished" podID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerID="c101f9bdf653ede0666117b92796b7a8f3855231edb0ac2305d73f3b5e528e36" exitCode=0
Oct 30 12:37:05.728834 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.728793    5837 generic.go:356] "Generic (PLEG): container finished" podID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerID="c00678908b8509631d5735197a35b3c60d7c5b2e9b6e1a1da46fc5043089fcc0" exitCode=0
Oct 30 12:37:05.729143 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.728828    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b16518-f7fa-4ddf-ab14-ace75b8d127b","Type":"ContainerDied","Data":"ba0cde52853918504e8e2b02c015fdf789a1964c1f93caa408dd1eeb878803ad"}
Oct 30 12:37:05.729143 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.728912    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b16518-f7fa-4ddf-ab14-ace75b8d127b","Type":"ContainerDied","Data":"01cb6c1f41abb8ee8d23626a841926d92a98a3abfa4fc270a32683230d6eda2f"}
Oct 30 12:37:05.729143 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.728958    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b16518-f7fa-4ddf-ab14-ace75b8d127b","Type":"ContainerDied","Data":"7d6311147738646a2b8099040a3efa4c265f0939d1d144f571e3ad1df8c54953"}
Oct 30 12:37:05.729143 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.728998    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b16518-f7fa-4ddf-ab14-ace75b8d127b","Type":"ContainerDied","Data":"c101f9bdf653ede0666117b92796b7a8f3855231edb0ac2305d73f3b5e528e36"}
Oct 30 12:37:05.729143 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:05.729038    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b16518-f7fa-4ddf-ab14-ace75b8d127b","Type":"ContainerDied","Data":"c00678908b8509631d5735197a35b3c60d7c5b2e9b6e1a1da46fc5043089fcc0"}
Oct 30 12:37:06.583328 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.583291    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.649138 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.649049    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-out\") pod \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") "
Oct 30 12:37:06.649138 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.649114    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") "
Oct 30 12:37:06.649375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.649145    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy\") pod \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") "
Oct 30 12:37:06.649375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.649173    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-main-tls\") pod \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") "
Oct 30 12:37:06.649375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.649212    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-web-config\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-web-config\") pod \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") "
Oct 30 12:37:06.649375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.649244    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-volume\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-volume\") pod \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") "
Oct 30 12:37:06.649724 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.649673    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-tls-assets\" (UniqueName: \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-tls-assets\") pod \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") "
Oct 30 12:37:06.649809 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.649735    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-kube-api-access-vqncb\" (UniqueName: \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-kube-api-access-vqncb\") pod \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") "
Oct 30 12:37:06.649809 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.649782    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-main-db\") pod \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") "
Oct 30 12:37:06.650070 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.649947    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-trusted-ca-bundle\") pod \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") "
Oct 30 12:37:06.650070 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.650044    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-metrics-client-ca\") pod \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") "
Oct 30 12:37:06.650175 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.650080    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-cluster-tls-config\") pod \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") "
Oct 30 12:37:06.650175 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.650115    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-web\") pod \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\" (UID: \"79b16518-f7fa-4ddf-ab14-ace75b8d127b\") "
Oct 30 12:37:06.650944 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.650613    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-main-db" pod "79b16518-f7fa-4ddf-ab14-ace75b8d127b" (UID: "79b16518-f7fa-4ddf-ab14-ace75b8d127b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:06.651386 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.651358    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-metrics-client-ca" pod "79b16518-f7fa-4ddf-ab14-ace75b8d127b" (UID: "79b16518-f7fa-4ddf-ab14-ace75b8d127b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:06.651856 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.651828    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-trusted-ca-bundle" pod "79b16518-f7fa-4ddf-ab14-ace75b8d127b" (UID: "79b16518-f7fa-4ddf-ab14-ace75b8d127b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:06.652805 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.652771    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-main-tls" pod "79b16518-f7fa-4ddf-ab14-ace75b8d127b" (UID: "79b16518-f7fa-4ddf-ab14-ace75b8d127b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:06.652903 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.652818    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-volume" pod "79b16518-f7fa-4ddf-ab14-ace75b8d127b" (UID: "79b16518-f7fa-4ddf-ab14-ace75b8d127b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:06.653916 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.653856    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-tls-assets" pod "79b16518-f7fa-4ddf-ab14-ace75b8d127b" (UID: "79b16518-f7fa-4ddf-ab14-ace75b8d127b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:06.654171 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.654136    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy" pod "79b16518-f7fa-4ddf-ab14-ace75b8d127b" (UID: "79b16518-f7fa-4ddf-ab14-ace75b8d127b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:06.654597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.654572    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-out" pod "79b16518-f7fa-4ddf-ab14-ace75b8d127b" (UID: "79b16518-f7fa-4ddf-ab14-ace75b8d127b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:06.654873 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.654832    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-metric" pod "79b16518-f7fa-4ddf-ab14-ace75b8d127b" (UID: "79b16518-f7fa-4ddf-ab14-ace75b8d127b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:06.654873 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.654844    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-kube-api-access-vqncb" pod "79b16518-f7fa-4ddf-ab14-ace75b8d127b" (UID: "79b16518-f7fa-4ddf-ab14-ace75b8d127b"). InnerVolumeSpecName "kube-api-access-vqncb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:06.655056 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.654922    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-web" pod "79b16518-f7fa-4ddf-ab14-ace75b8d127b" (UID: "79b16518-f7fa-4ddf-ab14-ace75b8d127b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:06.660355 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.660324    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-cluster-tls-config" pod "79b16518-f7fa-4ddf-ab14-ace75b8d127b" (UID: "79b16518-f7fa-4ddf-ab14-ace75b8d127b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:06.667513 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.667473    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-web-config" pod "79b16518-f7fa-4ddf-ab14-ace75b8d127b" (UID: "79b16518-f7fa-4ddf-ab14-ace75b8d127b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:06.751502 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.751462    5837 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:06.751502 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.751499    5837 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:06.751774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.751517    5837 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-main-tls\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:06.751774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.751536    5837 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-web-config\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:06.751774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.751549    5837 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:06.751774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.751562    5837 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-tls-assets\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:06.751774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.751575    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqncb\" (UniqueName: \"kubernetes.io/projected/79b16518-f7fa-4ddf-ab14-ace75b8d127b-kube-api-access-vqncb\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:06.751774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.751589    5837 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-main-db\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:06.751774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.751603    5837 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:06.751774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.751617    5837 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79b16518-f7fa-4ddf-ab14-ace75b8d127b-metrics-client-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:06.751774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.751630    5837 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-cluster-tls-config\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:06.751774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.751644    5837 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79b16518-f7fa-4ddf-ab14-ace75b8d127b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:06.751774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.751657    5837 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79b16518-f7fa-4ddf-ab14-ace75b8d127b-config-out\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:06.757167 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.757129    5837 generic.go:356] "Generic (PLEG): container finished" podID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerID="a22b4e94bbcd95169aed0f2e64ee27c20b883de077763946b807e9840fca4732" exitCode=0
Oct 30 12:37:06.757331 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.757212    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.757331 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.757212    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b16518-f7fa-4ddf-ab14-ace75b8d127b","Type":"ContainerDied","Data":"a22b4e94bbcd95169aed0f2e64ee27c20b883de077763946b807e9840fca4732"}
Oct 30 12:37:06.757331 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.757297    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b16518-f7fa-4ddf-ab14-ace75b8d127b","Type":"ContainerDied","Data":"32aa1f56c8be8accf7c779bf4f78e54aa34b11c7b8f4f503d9eaf693fb991370"}
Oct 30 12:37:06.757482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.757345    5837 scope.go:122] "RemoveContainer" containerID="ba0cde52853918504e8e2b02c015fdf789a1964c1f93caa408dd1eeb878803ad"
Oct 30 12:37:06.767313 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.767290    5837 scope.go:122] "RemoveContainer" containerID="01cb6c1f41abb8ee8d23626a841926d92a98a3abfa4fc270a32683230d6eda2f"
Oct 30 12:37:06.777969 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.777947    5837 scope.go:122] "RemoveContainer" containerID="7d6311147738646a2b8099040a3efa4c265f0939d1d144f571e3ad1df8c54953"
Oct 30 12:37:06.786266 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.786243    5837 scope.go:122] "RemoveContainer" containerID="a22b4e94bbcd95169aed0f2e64ee27c20b883de077763946b807e9840fca4732"
Oct 30 12:37:06.795112 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.795095    5837 scope.go:122] "RemoveContainer" containerID="c101f9bdf653ede0666117b92796b7a8f3855231edb0ac2305d73f3b5e528e36"
Oct 30 12:37:06.802841 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.802819    5837 scope.go:122] "RemoveContainer" containerID="c00678908b8509631d5735197a35b3c60d7c5b2e9b6e1a1da46fc5043089fcc0"
Oct 30 12:37:06.813104 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.813083    5837 scope.go:122] "RemoveContainer" containerID="efa532c5fe9cd30ec8a7deefcad58c09f85970b879b921fea96683d75c24e165"
Oct 30 12:37:06.820841 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.820820    5837 scope.go:122] "RemoveContainer" containerID="ba0cde52853918504e8e2b02c015fdf789a1964c1f93caa408dd1eeb878803ad"
Oct 30 12:37:06.821194 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:06.821169    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0cde52853918504e8e2b02c015fdf789a1964c1f93caa408dd1eeb878803ad\": container with ID starting with ba0cde52853918504e8e2b02c015fdf789a1964c1f93caa408dd1eeb878803ad not found: ID does not exist" containerID="ba0cde52853918504e8e2b02c015fdf789a1964c1f93caa408dd1eeb878803ad"
Oct 30 12:37:06.821264 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.821209    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0cde52853918504e8e2b02c015fdf789a1964c1f93caa408dd1eeb878803ad"} err="failed to get container status \"ba0cde52853918504e8e2b02c015fdf789a1964c1f93caa408dd1eeb878803ad\": rpc error: code = NotFound desc = could not find container \"ba0cde52853918504e8e2b02c015fdf789a1964c1f93caa408dd1eeb878803ad\": container with ID starting with ba0cde52853918504e8e2b02c015fdf789a1964c1f93caa408dd1eeb878803ad not found: ID does not exist"
Oct 30 12:37:06.821264 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.821257    5837 scope.go:122] "RemoveContainer" containerID="01cb6c1f41abb8ee8d23626a841926d92a98a3abfa4fc270a32683230d6eda2f"
Oct 30 12:37:06.821624 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:06.821602    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01cb6c1f41abb8ee8d23626a841926d92a98a3abfa4fc270a32683230d6eda2f\": container with ID starting with 01cb6c1f41abb8ee8d23626a841926d92a98a3abfa4fc270a32683230d6eda2f not found: ID does not exist" containerID="01cb6c1f41abb8ee8d23626a841926d92a98a3abfa4fc270a32683230d6eda2f"
Oct 30 12:37:06.821718 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.821630    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01cb6c1f41abb8ee8d23626a841926d92a98a3abfa4fc270a32683230d6eda2f"} err="failed to get container status \"01cb6c1f41abb8ee8d23626a841926d92a98a3abfa4fc270a32683230d6eda2f\": rpc error: code = NotFound desc = could not find container \"01cb6c1f41abb8ee8d23626a841926d92a98a3abfa4fc270a32683230d6eda2f\": container with ID starting with 01cb6c1f41abb8ee8d23626a841926d92a98a3abfa4fc270a32683230d6eda2f not found: ID does not exist"
Oct 30 12:37:06.821718 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.821649    5837 scope.go:122] "RemoveContainer" containerID="7d6311147738646a2b8099040a3efa4c265f0939d1d144f571e3ad1df8c54953"
Oct 30 12:37:06.822076 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:06.822003    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d6311147738646a2b8099040a3efa4c265f0939d1d144f571e3ad1df8c54953\": container with ID starting with 7d6311147738646a2b8099040a3efa4c265f0939d1d144f571e3ad1df8c54953 not found: ID does not exist" containerID="7d6311147738646a2b8099040a3efa4c265f0939d1d144f571e3ad1df8c54953"
Oct 30 12:37:06.822155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.822066    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d6311147738646a2b8099040a3efa4c265f0939d1d144f571e3ad1df8c54953"} err="failed to get container status \"7d6311147738646a2b8099040a3efa4c265f0939d1d144f571e3ad1df8c54953\": rpc error: code = NotFound desc = could not find container \"7d6311147738646a2b8099040a3efa4c265f0939d1d144f571e3ad1df8c54953\": container with ID starting with 7d6311147738646a2b8099040a3efa4c265f0939d1d144f571e3ad1df8c54953 not found: ID does not exist"
Oct 30 12:37:06.822155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.822095    5837 scope.go:122] "RemoveContainer" containerID="a22b4e94bbcd95169aed0f2e64ee27c20b883de077763946b807e9840fca4732"
Oct 30 12:37:06.822504 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:06.822479    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22b4e94bbcd95169aed0f2e64ee27c20b883de077763946b807e9840fca4732\": container with ID starting with a22b4e94bbcd95169aed0f2e64ee27c20b883de077763946b807e9840fca4732 not found: ID does not exist" containerID="a22b4e94bbcd95169aed0f2e64ee27c20b883de077763946b807e9840fca4732"
Oct 30 12:37:06.822619 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.822503    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22b4e94bbcd95169aed0f2e64ee27c20b883de077763946b807e9840fca4732"} err="failed to get container status \"a22b4e94bbcd95169aed0f2e64ee27c20b883de077763946b807e9840fca4732\": rpc error: code = NotFound desc = could not find container \"a22b4e94bbcd95169aed0f2e64ee27c20b883de077763946b807e9840fca4732\": container with ID starting with a22b4e94bbcd95169aed0f2e64ee27c20b883de077763946b807e9840fca4732 not found: ID does not exist"
Oct 30 12:37:06.822619 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.822518    5837 scope.go:122] "RemoveContainer" containerID="c101f9bdf653ede0666117b92796b7a8f3855231edb0ac2305d73f3b5e528e36"
Oct 30 12:37:06.822858 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:06.822828    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c101f9bdf653ede0666117b92796b7a8f3855231edb0ac2305d73f3b5e528e36\": container with ID starting with c101f9bdf653ede0666117b92796b7a8f3855231edb0ac2305d73f3b5e528e36 not found: ID does not exist" containerID="c101f9bdf653ede0666117b92796b7a8f3855231edb0ac2305d73f3b5e528e36"
Oct 30 12:37:06.822939 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.822858    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c101f9bdf653ede0666117b92796b7a8f3855231edb0ac2305d73f3b5e528e36"} err="failed to get container status \"c101f9bdf653ede0666117b92796b7a8f3855231edb0ac2305d73f3b5e528e36\": rpc error: code = NotFound desc = could not find container \"c101f9bdf653ede0666117b92796b7a8f3855231edb0ac2305d73f3b5e528e36\": container with ID starting with c101f9bdf653ede0666117b92796b7a8f3855231edb0ac2305d73f3b5e528e36 not found: ID does not exist"
Oct 30 12:37:06.822939 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.822877    5837 scope.go:122] "RemoveContainer" containerID="c00678908b8509631d5735197a35b3c60d7c5b2e9b6e1a1da46fc5043089fcc0"
Oct 30 12:37:06.823347 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:06.823324    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00678908b8509631d5735197a35b3c60d7c5b2e9b6e1a1da46fc5043089fcc0\": container with ID starting with c00678908b8509631d5735197a35b3c60d7c5b2e9b6e1a1da46fc5043089fcc0 not found: ID does not exist" containerID="c00678908b8509631d5735197a35b3c60d7c5b2e9b6e1a1da46fc5043089fcc0"
Oct 30 12:37:06.823479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.823357    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00678908b8509631d5735197a35b3c60d7c5b2e9b6e1a1da46fc5043089fcc0"} err="failed to get container status \"c00678908b8509631d5735197a35b3c60d7c5b2e9b6e1a1da46fc5043089fcc0\": rpc error: code = NotFound desc = could not find container \"c00678908b8509631d5735197a35b3c60d7c5b2e9b6e1a1da46fc5043089fcc0\": container with ID starting with c00678908b8509631d5735197a35b3c60d7c5b2e9b6e1a1da46fc5043089fcc0 not found: ID does not exist"
Oct 30 12:37:06.823479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.823378    5837 scope.go:122] "RemoveContainer" containerID="efa532c5fe9cd30ec8a7deefcad58c09f85970b879b921fea96683d75c24e165"
Oct 30 12:37:06.823700 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:06.823672    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa532c5fe9cd30ec8a7deefcad58c09f85970b879b921fea96683d75c24e165\": container with ID starting with efa532c5fe9cd30ec8a7deefcad58c09f85970b879b921fea96683d75c24e165 not found: ID does not exist" containerID="efa532c5fe9cd30ec8a7deefcad58c09f85970b879b921fea96683d75c24e165"
Oct 30 12:37:06.823817 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.823703    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa532c5fe9cd30ec8a7deefcad58c09f85970b879b921fea96683d75c24e165"} err="failed to get container status \"efa532c5fe9cd30ec8a7deefcad58c09f85970b879b921fea96683d75c24e165\": rpc error: code = NotFound desc = could not find container \"efa532c5fe9cd30ec8a7deefcad58c09f85970b879b921fea96683d75c24e165\": container with ID starting with efa532c5fe9cd30ec8a7deefcad58c09f85970b879b921fea96683d75c24e165 not found: ID does not exist"
Oct 30 12:37:06.823817 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.823766    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:37:06.834466 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.834441    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:37:06.886364 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886334    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:37:06.886664 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886653    5837 cpu_manager.go:414] "containerMap: removing container" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="config-reloader"
Oct 30 12:37:06.886709 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886666    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="config-reloader"
Oct 30 12:37:06.886709 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886684    5837 cpu_manager.go:414] "containerMap: removing container" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="prom-label-proxy"
Oct 30 12:37:06.886709 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886689    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="prom-label-proxy"
Oct 30 12:37:06.886709 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886698    5837 cpu_manager.go:414] "containerMap: removing container" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="kube-rbac-proxy-metric"
Oct 30 12:37:06.886709 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886703    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="kube-rbac-proxy-metric"
Oct 30 12:37:06.886709 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886710    5837 cpu_manager.go:414] "containerMap: removing container" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="kube-rbac-proxy"
Oct 30 12:37:06.886882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886715    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="kube-rbac-proxy"
Oct 30 12:37:06.886882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886725    5837 cpu_manager.go:414] "containerMap: removing container" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="alertmanager"
Oct 30 12:37:06.886882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886729    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="alertmanager"
Oct 30 12:37:06.886882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886736    5837 cpu_manager.go:414] "containerMap: removing container" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="init-config-reloader"
Oct 30 12:37:06.886882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886740    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="init-config-reloader"
Oct 30 12:37:06.886882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886749    5837 cpu_manager.go:414] "containerMap: removing container" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="kube-rbac-proxy-web"
Oct 30 12:37:06.886882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886754    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="kube-rbac-proxy-web"
Oct 30 12:37:06.886882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886810    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="prom-label-proxy"
Oct 30 12:37:06.886882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886820    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="kube-rbac-proxy-metric"
Oct 30 12:37:06.886882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886826    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="kube-rbac-proxy"
Oct 30 12:37:06.886882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886832    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="alertmanager"
Oct 30 12:37:06.886882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886838    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="init-config-reloader"
Oct 30 12:37:06.886882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886844    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="kube-rbac-proxy-web"
Oct 30 12:37:06.886882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.886852    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" containerName="config-reloader"
Oct 30 12:37:06.925966 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.925899    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.931483 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.931379    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:37:06.945981 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.945927    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 12:37:06.945981 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.945927    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 12:37:06.945981 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.945939    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 12:37:06.945981 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.945927    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 12:37:06.946284 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.946086    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dtxmh\""
Oct 30 12:37:06.946284 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.946125    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 12:37:06.946284 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.946158    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 12:37:06.946284 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.946163    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 12:37:06.946284 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.946230    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 12:37:06.953254 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.953228    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-web-config\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.953415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.953261    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xx95\" (UniqueName: \"kubernetes.io/projected/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-kube-api-access-5xx95\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.953415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.953299    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.953415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.953344    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-config-volume\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.953592 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.953470    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.953592 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.953510    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.953592 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.953582    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.953740 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.953612    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.953740 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.953684    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.953740 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.953714    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.953852 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.953744    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.953852 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.953768    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.953852 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.953789    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-config-out\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:06.957972 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:06.957953    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 12:37:07.055101 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.055059    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.055314 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.055110    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.055314 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.055149    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.055314 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.055176    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.055314 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.055205    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-config-out\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.055314 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.055248    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-web-config\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.055314 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.055267    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xx95\" (UniqueName: \"kubernetes.io/projected/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-kube-api-access-5xx95\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.055314 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.055311    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.055777 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.055334    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-config-volume\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.055777 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.055381    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.055777 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.055431    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.055777 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.055475    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.055777 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.055501    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.055777 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.055577    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.055777 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.055572    5837 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 12:37:07.056227 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.056207    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.056729 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.056526    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.058677 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.058654    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-config-out\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.058787 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.058756    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.059182 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.059163    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.059292 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.059268    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.059355 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.059301    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.059355 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.059328    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.059683 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.059666    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-web-config\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.059730 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.059713    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-config-volume\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.060435 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.060417    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.075139 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.075117    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xx95\" (UniqueName: \"kubernetes.io/projected/cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed-kube-api-access-5xx95\") pod \"alertmanager-main-0\" (UID: \"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.326171 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.326139    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:37:07.602507 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.602428    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b16518-f7fa-4ddf-ab14-ace75b8d127b" path="/var/lib/kubelet/pods/79b16518-f7fa-4ddf-ab14-ace75b8d127b/volumes"
Oct 30 12:37:07.673310 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.673272    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:37:07.686288 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:37:07.686251    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfac81cf_d52e_42f7_8bfd_5df2d9fdbaed.slice/crio-3b4ce234b5b357d6900d6758250777765a86b720eafea3d1ce1be3ccc32278d3 WatchSource:0}: Error finding container 3b4ce234b5b357d6900d6758250777765a86b720eafea3d1ce1be3ccc32278d3: Status 404 returned error can't find the container with id 3b4ce234b5b357d6900d6758250777765a86b720eafea3d1ce1be3ccc32278d3
Oct 30 12:37:07.763254 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.763226    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed","Type":"ContainerStarted","Data":"a2be7088a910dd9a8877b9b3ef443967b8df3520024e380103b4121da6377f21"}
Oct 30 12:37:07.763587 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:07.763285    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed","Type":"ContainerStarted","Data":"3b4ce234b5b357d6900d6758250777765a86b720eafea3d1ce1be3ccc32278d3"}
Oct 30 12:37:08.614712 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:08.614679    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n6d9z"
Oct 30 12:37:08.769085 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:08.769049    5837 generic.go:356] "Generic (PLEG): container finished" podID="cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed" containerID="a2be7088a910dd9a8877b9b3ef443967b8df3520024e380103b4121da6377f21" exitCode=0
Oct 30 12:37:08.769266 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:08.769137    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed","Type":"ContainerDied","Data":"a2be7088a910dd9a8877b9b3ef443967b8df3520024e380103b4121da6377f21"}
Oct 30 12:37:09.255831 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.255793    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7496d6d446-bnssg"]
Oct 30 12:37:09.258765 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.258743    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.275992 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.275965    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.276153 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.276003    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.276153 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.276036    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-metrics-client-ca\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.276153 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.276126    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wc87\" (UniqueName: \"kubernetes.io/projected/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-kube-api-access-9wc87\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.276269 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.276178    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-secret-telemeter-client\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.276269 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.276200    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-serving-certs-ca-bundle\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.276269 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.276222    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-telemeter-client-tls\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.276367 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.276275    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-federate-client-tls\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.277514 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.277498    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 12:37:09.277772 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.277755    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 12:37:09.278006 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.277993    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-76c4z\""
Oct 30 12:37:09.278065 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.277997    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 12:37:09.278248 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.278236    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 12:37:09.288987 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.288960    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7496d6d446-bnssg"]
Oct 30 12:37:09.327811 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.327780    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 12:37:09.340581 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.340556    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 12:37:09.377115 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.377080    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-metrics-client-ca\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.377115 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.377120    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wc87\" (UniqueName: \"kubernetes.io/projected/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-kube-api-access-9wc87\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.377361 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.377166    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-secret-telemeter-client\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.377361 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.377185    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-serving-certs-ca-bundle\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.377361 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.377203    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-telemeter-client-tls\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.377361 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.377217    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-federate-client-tls\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.377361 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.377255    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.377628 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.377535    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.378221 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.378189    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-serving-certs-ca-bundle\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.378221 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.378207    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-metrics-client-ca\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.378605 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.378580    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.379979 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.379953    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-secret-telemeter-client\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.380086 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.380071    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-federate-client-tls\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.380144 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.380099    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-telemeter-client-tls\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.380293 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.380269    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.397055 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.396999    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wc87\" (UniqueName: \"kubernetes.io/projected/b6118a33-9c2f-42e9-bd9a-27e350e26d7b-kube-api-access-9wc87\") pod \"telemeter-client-7496d6d446-bnssg\" (UID: \"b6118a33-9c2f-42e9-bd9a-27e350e26d7b\") " pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.571838 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.571807    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg"
Oct 30 12:37:09.776096 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.776056    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed","Type":"ContainerStarted","Data":"ce5dc1614f03899e2f546d79b33129a0759c0ce01e97e6d602317214ede93fdc"}
Oct 30 12:37:09.776623 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.776139    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed","Type":"ContainerStarted","Data":"9a42aba0cdfc352cb40eb6474e31788a7d258d12f85690c1d33c11bd442afe3d"}
Oct 30 12:37:09.776623 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.776184    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed","Type":"ContainerStarted","Data":"ebc8369967933fe3df1c0e6f82ea7e54fb5667c29e9e0501bc6f05401924fe7e"}
Oct 30 12:37:09.776623 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.776224    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed","Type":"ContainerStarted","Data":"a67a3947f8808757aacc4d1f6b80ff943dc731b0b6b85322205013f40dbdadad"}
Oct 30 12:37:09.776623 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.776271    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed","Type":"ContainerStarted","Data":"f7a6721e2ac1c46f7cdf8f4a782537732e855282f8ca1de538ec49d474cd9fff"}
Oct 30 12:37:09.776623 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.776306    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed","Type":"ContainerStarted","Data":"2162bc1d9c7ccb07177414bdbda54991b07004d3296b3a74b76223a450dc6a94"}
Oct 30 12:37:09.827895 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.827834    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.815145929 podStartE2EDuration="3.8278141s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.668171ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:09.827332391 +0000 UTC m=+18.351616609" watchObservedRunningTime="2026-10-30 12:37:09.8278141 +0000 UTC m=+18.352098318"
Oct 30 12:37:09.836120 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.836088    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:37:09.836488 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.836455    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="prometheus" containerID="cri-o://8ae65fb3e8b6f0bcb64a377fc741e4525198029cd4af0b6558b671f4245c41c1" gracePeriod=600
Oct 30 12:37:09.836671 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.836474    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="kube-rbac-proxy-web" containerID="cri-o://0142684f956bba4a10402aed864af544d0a20a30e765d31248d59ed93f7fc59e" gracePeriod=600
Oct 30 12:37:09.836671 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.836476    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://05fb4d7ee38cd3b282c0a2d30bf057d8b7682ee56635366dbb40d079bb99cae1" gracePeriod=600
Oct 30 12:37:09.836671 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.836585    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="thanos-sidecar" containerID="cri-o://244e96c496fa0b9dc5cb346fac30ac43d313a45a8ad502521fb1bfe113695c24" gracePeriod=600
Oct 30 12:37:09.836671 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.836609    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="config-reloader" containerID="cri-o://3aae098487a7adfbbbfc49d48f1ee310cf1c1a8b1d35a6f37c35859f72cf7370" gracePeriod=600
Oct 30 12:37:09.836855 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.836736    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="kube-rbac-proxy" containerID="cri-o://45b3f0de2d73c85f7e7c3aa864396d3c74d1347c5c17141e8ca51bcf772dbec9" gracePeriod=600
Oct 30 12:37:09.888782 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.888746    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7496d6d446-bnssg"]
Oct 30 12:37:09.891688 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:37:09.891654    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6118a33_9c2f_42e9_bd9a_27e350e26d7b.slice/crio-13101707c850d3a574a71627a4cebace714e97d794872d17ac0acd934b6ca90e WatchSource:0}: Error finding container 13101707c850d3a574a71627a4cebace714e97d794872d17ac0acd934b6ca90e: Status 404 returned error can't find the container with id 13101707c850d3a574a71627a4cebace714e97d794872d17ac0acd934b6ca90e
Oct 30 12:37:09.956070 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:09.956029    5837 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:37:10.781743 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:10.781705    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg" event={"ID":"b6118a33-9c2f-42e9-bd9a-27e350e26d7b","Type":"ContainerStarted","Data":"13101707c850d3a574a71627a4cebace714e97d794872d17ac0acd934b6ca90e"}
Oct 30 12:37:10.785509 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:10.785455    5837 generic.go:356] "Generic (PLEG): container finished" podID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerID="05fb4d7ee38cd3b282c0a2d30bf057d8b7682ee56635366dbb40d079bb99cae1" exitCode=0
Oct 30 12:37:10.785509 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:10.785481    5837 generic.go:356] "Generic (PLEG): container finished" podID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerID="45b3f0de2d73c85f7e7c3aa864396d3c74d1347c5c17141e8ca51bcf772dbec9" exitCode=0
Oct 30 12:37:10.785509 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:10.785493    5837 generic.go:356] "Generic (PLEG): container finished" podID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerID="244e96c496fa0b9dc5cb346fac30ac43d313a45a8ad502521fb1bfe113695c24" exitCode=0
Oct 30 12:37:10.785509 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:10.785502    5837 generic.go:356] "Generic (PLEG): container finished" podID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerID="3aae098487a7adfbbbfc49d48f1ee310cf1c1a8b1d35a6f37c35859f72cf7370" exitCode=0
Oct 30 12:37:10.785509 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:10.785512    5837 generic.go:356] "Generic (PLEG): container finished" podID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerID="8ae65fb3e8b6f0bcb64a377fc741e4525198029cd4af0b6558b671f4245c41c1" exitCode=0
Oct 30 12:37:10.786131 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:10.785527    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c1b5758-6ef0-452a-8ca7-c9457e30e60a","Type":"ContainerDied","Data":"05fb4d7ee38cd3b282c0a2d30bf057d8b7682ee56635366dbb40d079bb99cae1"}
Oct 30 12:37:10.786131 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:10.785608    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c1b5758-6ef0-452a-8ca7-c9457e30e60a","Type":"ContainerDied","Data":"45b3f0de2d73c85f7e7c3aa864396d3c74d1347c5c17141e8ca51bcf772dbec9"}
Oct 30 12:37:10.786131 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:10.785660    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c1b5758-6ef0-452a-8ca7-c9457e30e60a","Type":"ContainerDied","Data":"244e96c496fa0b9dc5cb346fac30ac43d313a45a8ad502521fb1bfe113695c24"}
Oct 30 12:37:10.786131 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:10.785702    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c1b5758-6ef0-452a-8ca7-c9457e30e60a","Type":"ContainerDied","Data":"3aae098487a7adfbbbfc49d48f1ee310cf1c1a8b1d35a6f37c35859f72cf7370"}
Oct 30 12:37:10.786131 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:10.785743    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c1b5758-6ef0-452a-8ca7-c9457e30e60a","Type":"ContainerDied","Data":"8ae65fb3e8b6f0bcb64a377fc741e4525198029cd4af0b6558b671f4245c41c1"}
Oct 30 12:37:11.112418 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.112367    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:11.193711 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.193676    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-db\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.193898 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.193730    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-serving-certs-ca-bundle\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.193898 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.193769    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-1\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.193898 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.193818    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.193898 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.193849    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-metrics-client-ca\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.193898 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.193884    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-0\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.194155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.193916    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-thanos-prometheus-http-client-file\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.194155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.193946    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-trusted-ca-bundle\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.194155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.193974    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config-out\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.194155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194011    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-web-config\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-web-config\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.194155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194047    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-kube-api-access-d57bx\" (UniqueName: \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-kube-api-access-d57bx\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.194155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194085    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-tls\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.194155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194126    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-2\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.194508 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194155    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-kubelet-serving-ca-bundle\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.194508 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194203    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.194508 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194237    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-tls-assets\" (UniqueName: \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-tls-assets\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.194508 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194281    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-grpc-tls\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.194508 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194314    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.194508 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194364    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-metrics-client-certs\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.194508 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194432    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-kube-rbac-proxy\") pod \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\" (UID: \"0c1b5758-6ef0-452a-8ca7-c9457e30e60a\") "
Oct 30 12:37:11.194817 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194624    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-serving-certs-ca-bundle" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:11.194817 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194647    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-db" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:11.194923 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194837    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-2" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:11.194923 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194847    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-1" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:11.195020 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194951    5837 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-db\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.195020 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194970    5837 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.195020 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.194985    5837 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.195167 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.195103    5837 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.195217 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.195173    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-metrics-client-ca" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:11.195262 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.195225    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-kubelet-serving-ca-bundle" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:11.196154 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.195878    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-0" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:11.197779 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.197747    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-trusted-ca-bundle" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:11.200464 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.200423    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-tls" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:11.202243 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.202200    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-kube-rbac-proxy" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:11.202365 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.202292    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:11.202811 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.202788    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-kube-api-access-d57bx" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "kube-api-access-d57bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:11.204663 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.204625    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-thanos-prometheus-http-client-file" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:11.206245 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.206127    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-metrics-client-certs" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:11.208178 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.207975    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-thanos-sidecar-tls" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:11.208178 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.208060    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config-out" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:11.208178 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.208091    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-grpc-tls" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:11.208178 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.208153    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-tls-assets" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:11.208990 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.208961    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-kube-rbac-proxy-web" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:11.224994 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.224958    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-web-config" pod "0c1b5758-6ef0-452a-8ca7-c9457e30e60a" (UID: "0c1b5758-6ef0-452a-8ca7-c9457e30e60a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:11.296194 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296151    5837 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-tls-assets\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.296194 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296182    5837 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-grpc-tls\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.296194 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296199    5837 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.296482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296214    5837 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-metrics-client-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.296482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296229    5837 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-kube-rbac-proxy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.296482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296243    5837 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.296482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296252    5837 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-metrics-client-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.296482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296264    5837 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.296482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296280    5837 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.296482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296293    5837 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.296482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296307    5837 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config-out\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.296482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296318    5837 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-web-config\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.296482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296332    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d57bx\" (UniqueName: \"kubernetes.io/projected/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-kube-api-access-d57bx\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.296482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296340    5837 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.296482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296349    5837 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.296482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.296357    5837 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c1b5758-6ef0-452a-8ca7-c9457e30e60a-config\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:37:11.792966 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.792934    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg" event={"ID":"b6118a33-9c2f-42e9-bd9a-27e350e26d7b","Type":"ContainerStarted","Data":"12b43719b16df23c7aad394abefcd39b8ec95263d80146d9f225f5c20e626cca"}
Oct 30 12:37:11.793341 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.792990    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg" event={"ID":"b6118a33-9c2f-42e9-bd9a-27e350e26d7b","Type":"ContainerStarted","Data":"764494a5bfd30e26a1ef624cf623879af588f4b43dca88bd1f298acbe801873b"}
Oct 30 12:37:11.796327 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.796299    5837 generic.go:356] "Generic (PLEG): container finished" podID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerID="0142684f956bba4a10402aed864af544d0a20a30e765d31248d59ed93f7fc59e" exitCode=0
Oct 30 12:37:11.796481 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.796335    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c1b5758-6ef0-452a-8ca7-c9457e30e60a","Type":"ContainerDied","Data":"0142684f956bba4a10402aed864af544d0a20a30e765d31248d59ed93f7fc59e"}
Oct 30 12:37:11.796481 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.796387    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c1b5758-6ef0-452a-8ca7-c9457e30e60a","Type":"ContainerDied","Data":"b0ddff1458fcd7794a111b51a4622cf59e3f3a08b52c2cf78ea14f6adbae9e64"}
Oct 30 12:37:11.796481 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.796417    5837 scope.go:122] "RemoveContainer" containerID="05fb4d7ee38cd3b282c0a2d30bf057d8b7682ee56635366dbb40d079bb99cae1"
Oct 30 12:37:11.796664 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.796418    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:11.804647 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.804618    5837 scope.go:122] "RemoveContainer" containerID="45b3f0de2d73c85f7e7c3aa864396d3c74d1347c5c17141e8ca51bcf772dbec9"
Oct 30 12:37:11.821196 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.821172    5837 scope.go:122] "RemoveContainer" containerID="0142684f956bba4a10402aed864af544d0a20a30e765d31248d59ed93f7fc59e"
Oct 30 12:37:11.828864 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.828834    5837 scope.go:122] "RemoveContainer" containerID="244e96c496fa0b9dc5cb346fac30ac43d313a45a8ad502521fb1bfe113695c24"
Oct 30 12:37:11.836474 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.836445    5837 scope.go:122] "RemoveContainer" containerID="3aae098487a7adfbbbfc49d48f1ee310cf1c1a8b1d35a6f37c35859f72cf7370"
Oct 30 12:37:11.843814 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.843793    5837 scope.go:122] "RemoveContainer" containerID="8ae65fb3e8b6f0bcb64a377fc741e4525198029cd4af0b6558b671f4245c41c1"
Oct 30 12:37:11.851883 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.851860    5837 scope.go:122] "RemoveContainer" containerID="d12dfc6d683790f1f80da3c4b6ef5105a72f9cea3026d150db56d8645f6d3206"
Oct 30 12:37:11.860190 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.860167    5837 scope.go:122] "RemoveContainer" containerID="05fb4d7ee38cd3b282c0a2d30bf057d8b7682ee56635366dbb40d079bb99cae1"
Oct 30 12:37:11.860576 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:11.860549    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05fb4d7ee38cd3b282c0a2d30bf057d8b7682ee56635366dbb40d079bb99cae1\": container with ID starting with 05fb4d7ee38cd3b282c0a2d30bf057d8b7682ee56635366dbb40d079bb99cae1 not found: ID does not exist" containerID="05fb4d7ee38cd3b282c0a2d30bf057d8b7682ee56635366dbb40d079bb99cae1"
Oct 30 12:37:11.860647 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.860592    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05fb4d7ee38cd3b282c0a2d30bf057d8b7682ee56635366dbb40d079bb99cae1"} err="failed to get container status \"05fb4d7ee38cd3b282c0a2d30bf057d8b7682ee56635366dbb40d079bb99cae1\": rpc error: code = NotFound desc = could not find container \"05fb4d7ee38cd3b282c0a2d30bf057d8b7682ee56635366dbb40d079bb99cae1\": container with ID starting with 05fb4d7ee38cd3b282c0a2d30bf057d8b7682ee56635366dbb40d079bb99cae1 not found: ID does not exist"
Oct 30 12:37:11.860647 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.860618    5837 scope.go:122] "RemoveContainer" containerID="45b3f0de2d73c85f7e7c3aa864396d3c74d1347c5c17141e8ca51bcf772dbec9"
Oct 30 12:37:11.860887 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:11.860871    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b3f0de2d73c85f7e7c3aa864396d3c74d1347c5c17141e8ca51bcf772dbec9\": container with ID starting with 45b3f0de2d73c85f7e7c3aa864396d3c74d1347c5c17141e8ca51bcf772dbec9 not found: ID does not exist" containerID="45b3f0de2d73c85f7e7c3aa864396d3c74d1347c5c17141e8ca51bcf772dbec9"
Oct 30 12:37:11.860933 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.860894    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b3f0de2d73c85f7e7c3aa864396d3c74d1347c5c17141e8ca51bcf772dbec9"} err="failed to get container status \"45b3f0de2d73c85f7e7c3aa864396d3c74d1347c5c17141e8ca51bcf772dbec9\": rpc error: code = NotFound desc = could not find container \"45b3f0de2d73c85f7e7c3aa864396d3c74d1347c5c17141e8ca51bcf772dbec9\": container with ID starting with 45b3f0de2d73c85f7e7c3aa864396d3c74d1347c5c17141e8ca51bcf772dbec9 not found: ID does not exist"
Oct 30 12:37:11.860933 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.860909    5837 scope.go:122] "RemoveContainer" containerID="0142684f956bba4a10402aed864af544d0a20a30e765d31248d59ed93f7fc59e"
Oct 30 12:37:11.861223 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:11.861186    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0142684f956bba4a10402aed864af544d0a20a30e765d31248d59ed93f7fc59e\": container with ID starting with 0142684f956bba4a10402aed864af544d0a20a30e765d31248d59ed93f7fc59e not found: ID does not exist" containerID="0142684f956bba4a10402aed864af544d0a20a30e765d31248d59ed93f7fc59e"
Oct 30 12:37:11.861324 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.861225    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0142684f956bba4a10402aed864af544d0a20a30e765d31248d59ed93f7fc59e"} err="failed to get container status \"0142684f956bba4a10402aed864af544d0a20a30e765d31248d59ed93f7fc59e\": rpc error: code = NotFound desc = could not find container \"0142684f956bba4a10402aed864af544d0a20a30e765d31248d59ed93f7fc59e\": container with ID starting with 0142684f956bba4a10402aed864af544d0a20a30e765d31248d59ed93f7fc59e not found: ID does not exist"
Oct 30 12:37:11.861324 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.861240    5837 scope.go:122] "RemoveContainer" containerID="244e96c496fa0b9dc5cb346fac30ac43d313a45a8ad502521fb1bfe113695c24"
Oct 30 12:37:11.861595 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:11.861577    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244e96c496fa0b9dc5cb346fac30ac43d313a45a8ad502521fb1bfe113695c24\": container with ID starting with 244e96c496fa0b9dc5cb346fac30ac43d313a45a8ad502521fb1bfe113695c24 not found: ID does not exist" containerID="244e96c496fa0b9dc5cb346fac30ac43d313a45a8ad502521fb1bfe113695c24"
Oct 30 12:37:11.861652 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.861600    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244e96c496fa0b9dc5cb346fac30ac43d313a45a8ad502521fb1bfe113695c24"} err="failed to get container status \"244e96c496fa0b9dc5cb346fac30ac43d313a45a8ad502521fb1bfe113695c24\": rpc error: code = NotFound desc = could not find container \"244e96c496fa0b9dc5cb346fac30ac43d313a45a8ad502521fb1bfe113695c24\": container with ID starting with 244e96c496fa0b9dc5cb346fac30ac43d313a45a8ad502521fb1bfe113695c24 not found: ID does not exist"
Oct 30 12:37:11.861652 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.861625    5837 scope.go:122] "RemoveContainer" containerID="3aae098487a7adfbbbfc49d48f1ee310cf1c1a8b1d35a6f37c35859f72cf7370"
Oct 30 12:37:11.861886 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:11.861868    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aae098487a7adfbbbfc49d48f1ee310cf1c1a8b1d35a6f37c35859f72cf7370\": container with ID starting with 3aae098487a7adfbbbfc49d48f1ee310cf1c1a8b1d35a6f37c35859f72cf7370 not found: ID does not exist" containerID="3aae098487a7adfbbbfc49d48f1ee310cf1c1a8b1d35a6f37c35859f72cf7370"
Oct 30 12:37:11.861946 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.861890    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aae098487a7adfbbbfc49d48f1ee310cf1c1a8b1d35a6f37c35859f72cf7370"} err="failed to get container status \"3aae098487a7adfbbbfc49d48f1ee310cf1c1a8b1d35a6f37c35859f72cf7370\": rpc error: code = NotFound desc = could not find container \"3aae098487a7adfbbbfc49d48f1ee310cf1c1a8b1d35a6f37c35859f72cf7370\": container with ID starting with 3aae098487a7adfbbbfc49d48f1ee310cf1c1a8b1d35a6f37c35859f72cf7370 not found: ID does not exist"
Oct 30 12:37:11.861946 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.861906    5837 scope.go:122] "RemoveContainer" containerID="8ae65fb3e8b6f0bcb64a377fc741e4525198029cd4af0b6558b671f4245c41c1"
Oct 30 12:37:11.862194 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:11.862171    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae65fb3e8b6f0bcb64a377fc741e4525198029cd4af0b6558b671f4245c41c1\": container with ID starting with 8ae65fb3e8b6f0bcb64a377fc741e4525198029cd4af0b6558b671f4245c41c1 not found: ID does not exist" containerID="8ae65fb3e8b6f0bcb64a377fc741e4525198029cd4af0b6558b671f4245c41c1"
Oct 30 12:37:11.862268 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.862202    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae65fb3e8b6f0bcb64a377fc741e4525198029cd4af0b6558b671f4245c41c1"} err="failed to get container status \"8ae65fb3e8b6f0bcb64a377fc741e4525198029cd4af0b6558b671f4245c41c1\": rpc error: code = NotFound desc = could not find container \"8ae65fb3e8b6f0bcb64a377fc741e4525198029cd4af0b6558b671f4245c41c1\": container with ID starting with 8ae65fb3e8b6f0bcb64a377fc741e4525198029cd4af0b6558b671f4245c41c1 not found: ID does not exist"
Oct 30 12:37:11.862268 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.862224    5837 scope.go:122] "RemoveContainer" containerID="d12dfc6d683790f1f80da3c4b6ef5105a72f9cea3026d150db56d8645f6d3206"
Oct 30 12:37:11.862520 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:11.862499    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12dfc6d683790f1f80da3c4b6ef5105a72f9cea3026d150db56d8645f6d3206\": container with ID starting with d12dfc6d683790f1f80da3c4b6ef5105a72f9cea3026d150db56d8645f6d3206 not found: ID does not exist" containerID="d12dfc6d683790f1f80da3c4b6ef5105a72f9cea3026d150db56d8645f6d3206"
Oct 30 12:37:11.862592 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.862524    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12dfc6d683790f1f80da3c4b6ef5105a72f9cea3026d150db56d8645f6d3206"} err="failed to get container status \"d12dfc6d683790f1f80da3c4b6ef5105a72f9cea3026d150db56d8645f6d3206\": rpc error: code = NotFound desc = could not find container \"d12dfc6d683790f1f80da3c4b6ef5105a72f9cea3026d150db56d8645f6d3206\": container with ID starting with d12dfc6d683790f1f80da3c4b6ef5105a72f9cea3026d150db56d8645f6d3206 not found: ID does not exist"
Oct 30 12:37:11.869347 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.869296    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:37:11.881220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.881195    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:37:11.942026 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.941992    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:37:11.942349 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942335    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="thanos-sidecar"
Oct 30 12:37:11.942447 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942350    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="thanos-sidecar"
Oct 30 12:37:11.942447 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942362    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="config-reloader"
Oct 30 12:37:11.942447 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942367    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="config-reloader"
Oct 30 12:37:11.942447 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942383    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="kube-rbac-proxy"
Oct 30 12:37:11.942447 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942405    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="kube-rbac-proxy"
Oct 30 12:37:11.942447 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942416    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="kube-rbac-proxy-thanos"
Oct 30 12:37:11.942447 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942424    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="kube-rbac-proxy-thanos"
Oct 30 12:37:11.942447 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942435    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="kube-rbac-proxy-web"
Oct 30 12:37:11.942447 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942441    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="kube-rbac-proxy-web"
Oct 30 12:37:11.942447 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942450    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="prometheus"
Oct 30 12:37:11.942741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942455    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="prometheus"
Oct 30 12:37:11.942741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942467    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="init-config-reloader"
Oct 30 12:37:11.942741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942471    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="init-config-reloader"
Oct 30 12:37:11.942741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942577    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="kube-rbac-proxy"
Oct 30 12:37:11.942741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942591    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="kube-rbac-proxy-thanos"
Oct 30 12:37:11.942741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942602    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="kube-rbac-proxy-web"
Oct 30 12:37:11.942741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942613    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="prometheus"
Oct 30 12:37:11.942741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942624    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="init-config-reloader"
Oct 30 12:37:11.942741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942632    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="thanos-sidecar"
Oct 30 12:37:11.942741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.942637    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" containerName="config-reloader"
Oct 30 12:37:11.946577 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.946560    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:11.961464 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.961436    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 12:37:11.961464 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.961449    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 12:37:11.961659 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.961621    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 12:37:11.962014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.961995    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 12:37:11.962014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.962009    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 12:37:11.962187 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.962019    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 12:37:11.962187 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.962142    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 12:37:11.962287 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.962199    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 12:37:11.962287 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.962203    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wbnrm\""
Oct 30 12:37:11.962287 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.962212    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 12:37:11.974549 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.974304    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 12:37:11.974549 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.974324    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1rld844cc7msd\""
Oct 30 12:37:11.974549 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.974337    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 12:37:11.974549 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.974347    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 12:37:11.974549 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.974352    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 12:37:11.974549 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.974519    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 12:37:11.975333 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:11.975311    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:37:12.000942 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.000906    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7vvv\" (UniqueName: \"kubernetes.io/projected/0a73f77c-5563-4a51-94ab-2febe951313c-kube-api-access-s7vvv\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.000948    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001021    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001065    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001217 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001142    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001217 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001170    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001217 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001196    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001305 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001244    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0a73f77c-5563-4a51-94ab-2febe951313c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001338 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001300    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a73f77c-5563-4a51-94ab-2febe951313c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001338 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001333    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001356    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001380    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001487 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001440    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001517 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001483    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001547 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001515    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001579 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001545    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001611 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001577    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a73f77c-5563-4a51-94ab-2febe951313c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001660 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001639    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001694 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001669    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.001727 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.001691    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-config\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.102597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.102546    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.102597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.102601    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.102800 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.102733    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0a73f77c-5563-4a51-94ab-2febe951313c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.102800 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.102794    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a73f77c-5563-4a51-94ab-2febe951313c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.102866 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.102819    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.102866 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.102854    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.102962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.102889    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.102962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.102919    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.102962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.102948    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.103106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.102986    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.103106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.103012    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.103106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.103041    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a73f77c-5563-4a51-94ab-2febe951313c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.103106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.103082    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.103303 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.103110    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.103303 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.103146    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-config\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.103303 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.103180    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0a73f77c-5563-4a51-94ab-2febe951313c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.103303 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.103188    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7vvv\" (UniqueName: \"kubernetes.io/projected/0a73f77c-5563-4a51-94ab-2febe951313c-kube-api-access-s7vvv\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.103303 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.103223    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.103303 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.103253    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.103303 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.103287    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.103687 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.103327    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.103988 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.103819    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.104721 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.104219    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.106603 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.106369    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.106603 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.106500    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.106603 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.106591    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a73f77c-5563-4a51-94ab-2febe951313c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.108147 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.106692    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.108147 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.106923    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.108147 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.106959    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a73f77c-5563-4a51-94ab-2febe951313c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.108147 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.107130    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.108147 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.107693    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.108147 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.107886    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0a73f77c-5563-4a51-94ab-2febe951313c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.108876 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.108823    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.110426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.109566    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.110426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.109782    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.110426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.110074    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-config\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.112643 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.110748    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.112643 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.110769    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.114268 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.114245    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a73f77c-5563-4a51-94ab-2febe951313c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.123667 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.123615    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7vvv\" (UniqueName: \"kubernetes.io/projected/0a73f77c-5563-4a51-94ab-2febe951313c-kube-api-access-s7vvv\") pod \"prometheus-k8s-0\" (UID: \"0a73f77c-5563-4a51-94ab-2febe951313c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.266803 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.266765    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:12.512424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.512382    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9trzv"
Oct 30 12:37:12.588004 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.587947    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:37:12.590280 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:37:12.590253    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a73f77c_5563_4a51_94ab_2febe951313c.slice/crio-69a0d7fa507f62a0e9b84b9b1188cdcc2fbeb3a419d9f5c0884e04baab1372b5 WatchSource:0}: Error finding container 69a0d7fa507f62a0e9b84b9b1188cdcc2fbeb3a419d9f5c0884e04baab1372b5: Status 404 returned error can't find the container with id 69a0d7fa507f62a0e9b84b9b1188cdcc2fbeb3a419d9f5c0884e04baab1372b5
Oct 30 12:37:12.803074 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.802977    5837 generic.go:356] "Generic (PLEG): container finished" podID="0a73f77c-5563-4a51-94ab-2febe951313c" containerID="06f740341b95c310277e4e48e62f49c48183b3f78b6bdc64eebe5ae5e882b3bf" exitCode=0
Oct 30 12:37:12.803573 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.803066    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a73f77c-5563-4a51-94ab-2febe951313c","Type":"ContainerDied","Data":"06f740341b95c310277e4e48e62f49c48183b3f78b6bdc64eebe5ae5e882b3bf"}
Oct 30 12:37:12.803573 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.803156    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a73f77c-5563-4a51-94ab-2febe951313c","Type":"ContainerStarted","Data":"69a0d7fa507f62a0e9b84b9b1188cdcc2fbeb3a419d9f5c0884e04baab1372b5"}
Oct 30 12:37:12.805563 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.805427    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg" event={"ID":"b6118a33-9c2f-42e9-bd9a-27e350e26d7b","Type":"ContainerStarted","Data":"02e5fcf2a76aff1d63b1f540269cec09e084f1454347e3d333dc0eefa75990be"}
Oct 30 12:37:12.895195 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:12.895135    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7496d6d446-bnssg" podStartSLOduration=2.200687545 podStartE2EDuration="3.895115196s" totalImagesPullingTime="1.694427651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:12.894378304 +0000 UTC m=+21.418662524" watchObservedRunningTime="2026-10-30 12:37:12.895115196 +0000 UTC m=+21.419399416"
Oct 30 12:37:13.602618 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:13.602587    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c1b5758-6ef0-452a-8ca7-c9457e30e60a" path="/var/lib/kubelet/pods/0c1b5758-6ef0-452a-8ca7-c9457e30e60a/volumes"
Oct 30 12:37:13.813328 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:13.813289    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a73f77c-5563-4a51-94ab-2febe951313c","Type":"ContainerStarted","Data":"ab72dccc2e18fbad2edaf38c0e937bcd21136e9f82ffb87d7a96ef3c09cc6cb5"}
Oct 30 12:37:13.813809 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:13.813376    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a73f77c-5563-4a51-94ab-2febe951313c","Type":"ContainerStarted","Data":"8dbb7e2f4a1d2c5797379b65bbcaba09aa90d32d76e620379e5127a87e86881a"}
Oct 30 12:37:13.813809 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:13.813436    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a73f77c-5563-4a51-94ab-2febe951313c","Type":"ContainerStarted","Data":"35815e961b1b16e4413817ceac5a6acb5795f54ee6fde128066d63c01b725a27"}
Oct 30 12:37:13.813809 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:13.813476    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a73f77c-5563-4a51-94ab-2febe951313c","Type":"ContainerStarted","Data":"b80c8edabd82d9036534ea899cd16aa711c39ea0dbb55806970384ed23efb70d"}
Oct 30 12:37:13.813809 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:13.813521    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a73f77c-5563-4a51-94ab-2febe951313c","Type":"ContainerStarted","Data":"c2b9dcc1c46bef866b7466fd37d857d82baee637f97e61b4c3bdc27245982cb6"}
Oct 30 12:37:13.813809 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:13.813557    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a73f77c-5563-4a51-94ab-2febe951313c","Type":"ContainerStarted","Data":"90436155ca0ae103fdaefec33f81c733fa15eee2091513d6024c39394024795e"}
Oct 30 12:37:13.876614 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:13.876557    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.85853741 podStartE2EDuration="2.876542195s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.004785ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:13.875788257 +0000 UTC m=+22.400072476" watchObservedRunningTime="2026-10-30 12:37:13.876542195 +0000 UTC m=+22.400826393"
Oct 30 12:37:14.871455 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:14.871414    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:14.888357 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:14.888327    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:15.096678 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:15.096636    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:37:15.822293 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:15.822268    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:15.838420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:15.838370    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:37:20.593344 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:37:20.593316    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-chqf2"
Oct 30 12:37:45.094271 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:37:45.094228    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:38:15.093262 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:38:15.093217    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:38:45.094163 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:38:45.094126    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:39:15.093768 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:39:15.093730    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:39:45.094790 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:39:45.094754    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:39:59.332089 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.332052    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"]
Oct 30 12:39:59.335898 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.335877    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.370855 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.370824    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"]
Oct 30 12:39:59.427199 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.427152    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 12:39:59.427199 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.427165    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 12:39:59.427451 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.427214    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 12:39:59.427451 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.427278    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-r2gbl\""
Oct 30 12:39:59.445582 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.445561    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 12:39:59.445582 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.445573    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 12:39:59.473028 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.472997    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/23f2d01a-5187-429e-bfb2-88fc1dc560fc-manifests\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.473121 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.473035    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqzx\" (UniqueName: \"kubernetes.io/projected/23f2d01a-5187-429e-bfb2-88fc1dc560fc-kube-api-access-jpqzx\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.473163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.473126    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/23f2d01a-5187-429e-bfb2-88fc1dc560fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.473197 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.473177    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/23f2d01a-5187-429e-bfb2-88fc1dc560fc-metrics-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.473229 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.473195    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23f2d01a-5187-429e-bfb2-88fc1dc560fc-tmp\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.473285 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.473270    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23f2d01a-5187-429e-bfb2-88fc1dc560fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.574671 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.574634    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23f2d01a-5187-429e-bfb2-88fc1dc560fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.574671 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.574679    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/23f2d01a-5187-429e-bfb2-88fc1dc560fc-manifests\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.574936 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.574703    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqzx\" (UniqueName: \"kubernetes.io/projected/23f2d01a-5187-429e-bfb2-88fc1dc560fc-kube-api-access-jpqzx\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.574936 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.574828    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/23f2d01a-5187-429e-bfb2-88fc1dc560fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.574936 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.574924    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/23f2d01a-5187-429e-bfb2-88fc1dc560fc-metrics-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.575102 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.574951    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23f2d01a-5187-429e-bfb2-88fc1dc560fc-tmp\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.575153 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.575105    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/23f2d01a-5187-429e-bfb2-88fc1dc560fc-manifests\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.577255 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.577220    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23f2d01a-5187-429e-bfb2-88fc1dc560fc-tmp\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.577469 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.577450    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23f2d01a-5187-429e-bfb2-88fc1dc560fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.577555 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.577493    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/23f2d01a-5187-429e-bfb2-88fc1dc560fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.577555 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.577523    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/23f2d01a-5187-429e-bfb2-88fc1dc560fc-metrics-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.602484 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.602417    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqzx\" (UniqueName: \"kubernetes.io/projected/23f2d01a-5187-429e-bfb2-88fc1dc560fc-kube-api-access-jpqzx\") pod \"opendatahub-operator-controller-manager-786cc47469-vr26k\" (UID: \"23f2d01a-5187-429e-bfb2-88fc1dc560fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.651282 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.651249    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:39:59.974017 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.973934    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-snzms"]
Oct 30 12:39:59.977424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:39:59.977387    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms"
Oct 30 12:40:00.000631 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:00.000599    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-r5rtx\""
Oct 30 12:40:00.001059 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:00.000993    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 12:40:00.001121 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:00.001080    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 12:40:00.001583 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:00.001560    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"]
Oct 30 12:40:00.005759 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:00.005724    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-snzms"]
Oct 30 12:40:00.006285 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:40:00.006257    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f2d01a_5187_429e_bfb2_88fc1dc560fc.slice/crio-5df89be2b4238c0c2638a8a3805c897a75df60c4f85c6d084805e8cdb447a280 WatchSource:0}: Error finding container 5df89be2b4238c0c2638a8a3805c897a75df60c4f85c6d084805e8cdb447a280: Status 404 returned error can't find the container with id 5df89be2b4238c0c2638a8a3805c897a75df60c4f85c6d084805e8cdb447a280
Oct 30 12:40:00.079322 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:00.079282    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7qlj\" (UniqueName: \"kubernetes.io/projected/b35e847c-8794-4355-8d45-f474df36feee-kube-api-access-s7qlj\") pod \"cert-manager-webhook-5dbbbf75cc-snzms\" (UID: \"b35e847c-8794-4355-8d45-f474df36feee\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms"
Oct 30 12:40:00.079543 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:00.079331    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b35e847c-8794-4355-8d45-f474df36feee-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-snzms\" (UID: \"b35e847c-8794-4355-8d45-f474df36feee\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms"
Oct 30 12:40:00.180326 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:00.180281    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7qlj\" (UniqueName: \"kubernetes.io/projected/b35e847c-8794-4355-8d45-f474df36feee-kube-api-access-s7qlj\") pod \"cert-manager-webhook-5dbbbf75cc-snzms\" (UID: \"b35e847c-8794-4355-8d45-f474df36feee\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms"
Oct 30 12:40:00.180530 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:00.180341    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b35e847c-8794-4355-8d45-f474df36feee-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-snzms\" (UID: \"b35e847c-8794-4355-8d45-f474df36feee\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms"
Oct 30 12:40:00.220185 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:00.220155    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b35e847c-8794-4355-8d45-f474df36feee-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-snzms\" (UID: \"b35e847c-8794-4355-8d45-f474df36feee\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms"
Oct 30 12:40:00.220328 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:00.220281    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7qlj\" (UniqueName: \"kubernetes.io/projected/b35e847c-8794-4355-8d45-f474df36feee-kube-api-access-s7qlj\") pod \"cert-manager-webhook-5dbbbf75cc-snzms\" (UID: \"b35e847c-8794-4355-8d45-f474df36feee\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms"
Oct 30 12:40:00.297057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:00.297028    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms"
Oct 30 12:40:00.451774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:00.451735    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k" event={"ID":"23f2d01a-5187-429e-bfb2-88fc1dc560fc","Type":"ContainerStarted","Data":"5df89be2b4238c0c2638a8a3805c897a75df60c4f85c6d084805e8cdb447a280"}
Oct 30 12:40:00.600186 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:00.600109    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-snzms"]
Oct 30 12:40:00.604843 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:40:00.604810    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb35e847c_8794_4355_8d45_f474df36feee.slice/crio-d01ffe422e9aea962ec3c4a75ba4144bef4b1139bb1a5d4f64f00dba8d52179e WatchSource:0}: Error finding container d01ffe422e9aea962ec3c4a75ba4144bef4b1139bb1a5d4f64f00dba8d52179e: Status 404 returned error can't find the container with id d01ffe422e9aea962ec3c4a75ba4144bef4b1139bb1a5d4f64f00dba8d52179e
Oct 30 12:40:01.461103 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:01.461058    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms" event={"ID":"b35e847c-8794-4355-8d45-f474df36feee","Type":"ContainerStarted","Data":"d01ffe422e9aea962ec3c4a75ba4144bef4b1139bb1a5d4f64f00dba8d52179e"}
Oct 30 12:40:03.472720 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:03.472677    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k" event={"ID":"23f2d01a-5187-429e-bfb2-88fc1dc560fc","Type":"ContainerStarted","Data":"ecdcfafe01fe491f398a324fb0a7596c9eef9a250c13a4b655bdda65c55f865f"}
Oct 30 12:40:04.478561 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:04.478535    5837 generic.go:356] "Generic (PLEG): container finished" podID="23f2d01a-5187-429e-bfb2-88fc1dc560fc" containerID="ecdcfafe01fe491f398a324fb0a7596c9eef9a250c13a4b655bdda65c55f865f" exitCode=0
Oct 30 12:40:04.478823 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:04.478623    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k" event={"ID":"23f2d01a-5187-429e-bfb2-88fc1dc560fc","Type":"ContainerDied","Data":"ecdcfafe01fe491f398a324fb0a7596c9eef9a250c13a4b655bdda65c55f865f"}
Oct 30 12:40:05.484543 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:05.484512    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k" event={"ID":"23f2d01a-5187-429e-bfb2-88fc1dc560fc","Type":"ContainerStarted","Data":"0886b6e87634049f818b8efb0e1f851494209fb485077efb300550b09ad4c324"}
Oct 30 12:40:05.485091 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:05.484629    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:40:05.486187 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:05.486166    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms" event={"ID":"b35e847c-8794-4355-8d45-f474df36feee","Type":"ContainerStarted","Data":"a09e8e96141d61e25f497bb84bc5c47ade3622479c38eec2c0bc79fee55f6c03"}
Oct 30 12:40:05.486285 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:05.486271    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms"
Oct 30 12:40:05.616641 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:05.616582    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k" podStartSLOduration=2.3347842180000002 podStartE2EDuration="6.616565216s" totalImagesPullingTime="3.660791576s" totalInitContainerRuntime="620.989422ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:05.604278242 +0000 UTC m=+194.128562463" watchObservedRunningTime="2026-10-30 12:40:05.616565216 +0000 UTC m=+194.140849435"
Oct 30 12:40:05.670208 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:05.670154    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms" podStartSLOduration=2.798531992 podStartE2EDuration="6.670138687s" totalImagesPullingTime="3.871606695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:05.66976509 +0000 UTC m=+194.194049310" watchObservedRunningTime="2026-10-30 12:40:05.670138687 +0000 UTC m=+194.194422906"
Oct 30 12:40:08.477261 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:08.477226    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fb2pt"]
Oct 30 12:40:08.480748 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:08.480731    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fb2pt"
Oct 30 12:40:08.508731 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:08.508699    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:40:08.508731 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:08.508699    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 12:40:08.508935 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:08.508891    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-rqc82\""
Oct 30 12:40:08.510197 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:08.510175    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fb2pt"]
Oct 30 12:40:08.566005 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:08.565974    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmcs4\" (UniqueName: \"kubernetes.io/projected/978ebc73-4c48-4b11-8aa3-ef8adbdea1c1-kube-api-access-kmcs4\") pod \"jobset-operator-674cd59dc5-fb2pt\" (UID: \"978ebc73-4c48-4b11-8aa3-ef8adbdea1c1\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fb2pt"
Oct 30 12:40:08.566188 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:08.566018    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/978ebc73-4c48-4b11-8aa3-ef8adbdea1c1-tmp\") pod \"jobset-operator-674cd59dc5-fb2pt\" (UID: \"978ebc73-4c48-4b11-8aa3-ef8adbdea1c1\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fb2pt"
Oct 30 12:40:08.667179 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:08.667138    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmcs4\" (UniqueName: \"kubernetes.io/projected/978ebc73-4c48-4b11-8aa3-ef8adbdea1c1-kube-api-access-kmcs4\") pod \"jobset-operator-674cd59dc5-fb2pt\" (UID: \"978ebc73-4c48-4b11-8aa3-ef8adbdea1c1\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fb2pt"
Oct 30 12:40:08.667179 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:08.667185    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/978ebc73-4c48-4b11-8aa3-ef8adbdea1c1-tmp\") pod \"jobset-operator-674cd59dc5-fb2pt\" (UID: \"978ebc73-4c48-4b11-8aa3-ef8adbdea1c1\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fb2pt"
Oct 30 12:40:08.667560 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:08.667532    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/978ebc73-4c48-4b11-8aa3-ef8adbdea1c1-tmp\") pod \"jobset-operator-674cd59dc5-fb2pt\" (UID: \"978ebc73-4c48-4b11-8aa3-ef8adbdea1c1\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fb2pt"
Oct 30 12:40:08.704174 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:08.704140    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmcs4\" (UniqueName: \"kubernetes.io/projected/978ebc73-4c48-4b11-8aa3-ef8adbdea1c1-kube-api-access-kmcs4\") pod \"jobset-operator-674cd59dc5-fb2pt\" (UID: \"978ebc73-4c48-4b11-8aa3-ef8adbdea1c1\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fb2pt"
Oct 30 12:40:08.791982 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:08.791943    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fb2pt"
Oct 30 12:40:09.097711 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:09.097627    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fb2pt"]
Oct 30 12:40:09.100863 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:40:09.100834    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod978ebc73_4c48_4b11_8aa3_ef8adbdea1c1.slice/crio-f8eff92e39cf2371df6f579f598062d938674d6f3a37fbe4e707f77a734509f6 WatchSource:0}: Error finding container f8eff92e39cf2371df6f579f598062d938674d6f3a37fbe4e707f77a734509f6: Status 404 returned error can't find the container with id f8eff92e39cf2371df6f579f598062d938674d6f3a37fbe4e707f77a734509f6
Oct 30 12:40:09.504746 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:09.504714    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fb2pt" event={"ID":"978ebc73-4c48-4b11-8aa3-ef8adbdea1c1","Type":"ContainerStarted","Data":"f8eff92e39cf2371df6f579f598062d938674d6f3a37fbe4e707f77a734509f6"}
Oct 30 12:40:10.257057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:10.257017    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-xw6zk"]
Oct 30 12:40:10.264669 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:10.264645    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-xw6zk"
Oct 30 12:40:10.277248 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:10.277226    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-xw6zk"]
Oct 30 12:40:10.288206 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:10.288177    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-vf58r\""
Oct 30 12:40:10.383527 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:10.383486    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4e9c69a-f988-41f9-9f91-7d6078fb82cd-bound-sa-token\") pod \"cert-manager-597b95cb84-xw6zk\" (UID: \"f4e9c69a-f988-41f9-9f91-7d6078fb82cd\") " pod="cert-manager/cert-manager-597b95cb84-xw6zk"
Oct 30 12:40:10.383749 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:10.383634    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbldp\" (UniqueName: \"kubernetes.io/projected/f4e9c69a-f988-41f9-9f91-7d6078fb82cd-kube-api-access-kbldp\") pod \"cert-manager-597b95cb84-xw6zk\" (UID: \"f4e9c69a-f988-41f9-9f91-7d6078fb82cd\") " pod="cert-manager/cert-manager-597b95cb84-xw6zk"
Oct 30 12:40:10.485048 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:10.485007    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbldp\" (UniqueName: \"kubernetes.io/projected/f4e9c69a-f988-41f9-9f91-7d6078fb82cd-kube-api-access-kbldp\") pod \"cert-manager-597b95cb84-xw6zk\" (UID: \"f4e9c69a-f988-41f9-9f91-7d6078fb82cd\") " pod="cert-manager/cert-manager-597b95cb84-xw6zk"
Oct 30 12:40:10.485237 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:10.485088    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4e9c69a-f988-41f9-9f91-7d6078fb82cd-bound-sa-token\") pod \"cert-manager-597b95cb84-xw6zk\" (UID: \"f4e9c69a-f988-41f9-9f91-7d6078fb82cd\") " pod="cert-manager/cert-manager-597b95cb84-xw6zk"
Oct 30 12:40:10.506956 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:10.506923    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4e9c69a-f988-41f9-9f91-7d6078fb82cd-bound-sa-token\") pod \"cert-manager-597b95cb84-xw6zk\" (UID: \"f4e9c69a-f988-41f9-9f91-7d6078fb82cd\") " pod="cert-manager/cert-manager-597b95cb84-xw6zk"
Oct 30 12:40:10.507502 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:10.507043    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbldp\" (UniqueName: \"kubernetes.io/projected/f4e9c69a-f988-41f9-9f91-7d6078fb82cd-kube-api-access-kbldp\") pod \"cert-manager-597b95cb84-xw6zk\" (UID: \"f4e9c69a-f988-41f9-9f91-7d6078fb82cd\") " pod="cert-manager/cert-manager-597b95cb84-xw6zk"
Oct 30 12:40:10.577163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:10.577130    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-xw6zk"
Oct 30 12:40:10.873121 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:10.873097    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-xw6zk"]
Oct 30 12:40:10.875929 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:40:10.875898    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4e9c69a_f988_41f9_9f91_7d6078fb82cd.slice/crio-f7693cf57104e494ff4a202b4d3d0960b2611d3a8787837bd5e6872c42ddab3c WatchSource:0}: Error finding container f7693cf57104e494ff4a202b4d3d0960b2611d3a8787837bd5e6872c42ddab3c: Status 404 returned error can't find the container with id f7693cf57104e494ff4a202b4d3d0960b2611d3a8787837bd5e6872c42ddab3c
Oct 30 12:40:11.493220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:11.493190    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms"
Oct 30 12:40:11.515622 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:11.515590    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-xw6zk" event={"ID":"f4e9c69a-f988-41f9-9f91-7d6078fb82cd","Type":"ContainerStarted","Data":"3127d66abc53aebf10c7a565f80a197f32db7cb7194003eadb346ff36f8c3189"}
Oct 30 12:40:11.516079 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:11.515631    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-xw6zk" event={"ID":"f4e9c69a-f988-41f9-9f91-7d6078fb82cd","Type":"ContainerStarted","Data":"f7693cf57104e494ff4a202b4d3d0960b2611d3a8787837bd5e6872c42ddab3c"}
Oct 30 12:40:11.585562 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:11.585512    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-xw6zk" podStartSLOduration=1.5854990610000002 podStartE2EDuration="1.585499061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:11.584327581 +0000 UTC m=+200.108611819" watchObservedRunningTime="2026-10-30 12:40:11.585499061 +0000 UTC m=+200.109783280"
Oct 30 12:40:15.093878 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:40:15.093841    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:40:16.492809 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:16.492782    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-vr26k"
Oct 30 12:40:19.552353 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:19.552314    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fb2pt" event={"ID":"978ebc73-4c48-4b11-8aa3-ef8adbdea1c1","Type":"ContainerStarted","Data":"65bbdf2952dba8fca895ae7f6a7cf13d99288192bbd3e8603e20cac88aa6b690"}
Oct 30 12:40:24.738837 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:24.738776    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fb2pt" podStartSLOduration=6.704611583 podStartE2EDuration="16.738759758s" totalImagesPullingTime="10.034148175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:19.605646322 +0000 UTC m=+208.129930543" watchObservedRunningTime="2026-10-30 12:40:24.738759758 +0000 UTC m=+213.263043976"
Oct 30 12:40:24.739241 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:24.739176    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-68669df9c4-2nng6"]
Oct 30 12:40:24.741959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:24.741939    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-2nng6"
Oct 30 12:40:24.759169 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:24.759136    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-47skr\""
Oct 30 12:40:24.770957 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:24.770931    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 12:40:24.771102 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:24.770979    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 12:40:24.771906 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:24.771887    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-68669df9c4-2nng6"]
Oct 30 12:40:24.821325 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:24.821286    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbt6j\" (UniqueName: \"kubernetes.io/projected/17d543a6-06a9-46ac-9b53-e43b0c0b34f5-kube-api-access-nbt6j\") pod \"limitador-operator-controller-manager-68669df9c4-2nng6\" (UID: \"17d543a6-06a9-46ac-9b53-e43b0c0b34f5\") " pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-2nng6"
Oct 30 12:40:24.921813 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:24.921763    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbt6j\" (UniqueName: \"kubernetes.io/projected/17d543a6-06a9-46ac-9b53-e43b0c0b34f5-kube-api-access-nbt6j\") pod \"limitador-operator-controller-manager-68669df9c4-2nng6\" (UID: \"17d543a6-06a9-46ac-9b53-e43b0c0b34f5\") " pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-2nng6"
Oct 30 12:40:24.941340 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:24.941312    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbt6j\" (UniqueName: \"kubernetes.io/projected/17d543a6-06a9-46ac-9b53-e43b0c0b34f5-kube-api-access-nbt6j\") pod \"limitador-operator-controller-manager-68669df9c4-2nng6\" (UID: \"17d543a6-06a9-46ac-9b53-e43b0c0b34f5\") " pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-2nng6"
Oct 30 12:40:25.053501 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:25.053414    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-2nng6"
Oct 30 12:40:25.352286 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:25.352122    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-68669df9c4-2nng6"]
Oct 30 12:40:25.355085 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:40:25.355049    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d543a6_06a9_46ac_9b53_e43b0c0b34f5.slice/crio-a972c3e2eaa989ac50853d263f7a6c0097d1a1b36dd4051c515e77655baf63d2 WatchSource:0}: Error finding container a972c3e2eaa989ac50853d263f7a6c0097d1a1b36dd4051c515e77655baf63d2: Status 404 returned error can't find the container with id a972c3e2eaa989ac50853d263f7a6c0097d1a1b36dd4051c515e77655baf63d2
Oct 30 12:40:25.578413 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:25.578356    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-2nng6" event={"ID":"17d543a6-06a9-46ac-9b53-e43b0c0b34f5","Type":"ContainerStarted","Data":"a972c3e2eaa989ac50853d263f7a6c0097d1a1b36dd4051c515e77655baf63d2"}
Oct 30 12:40:25.696828 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:25.696751    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-tvp9w"]
Oct 30 12:40:25.700126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:25.700104    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-tvp9w"
Oct 30 12:40:25.714329 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:25.714305    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5twgk\""
Oct 30 12:40:25.726202 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:25.726176    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 12:40:25.727176 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:25.727153    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-tvp9w"]
Oct 30 12:40:25.728648 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:25.728619    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c9068e7-5aea-467f-ad19-b6421524c6de-plugin-serving-cert\") pod \"kuadrant-console-plugin-7bbc6bb9d7-tvp9w\" (UID: \"6c9068e7-5aea-467f-ad19-b6421524c6de\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-tvp9w"
Oct 30 12:40:25.728761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:25.728740    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddr9r\" (UniqueName: \"kubernetes.io/projected/6c9068e7-5aea-467f-ad19-b6421524c6de-kube-api-access-ddr9r\") pod \"kuadrant-console-plugin-7bbc6bb9d7-tvp9w\" (UID: \"6c9068e7-5aea-467f-ad19-b6421524c6de\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-tvp9w"
Oct 30 12:40:25.830068 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:25.830025    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c9068e7-5aea-467f-ad19-b6421524c6de-plugin-serving-cert\") pod \"kuadrant-console-plugin-7bbc6bb9d7-tvp9w\" (UID: \"6c9068e7-5aea-467f-ad19-b6421524c6de\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-tvp9w"
Oct 30 12:40:25.830571 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:25.830096    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddr9r\" (UniqueName: \"kubernetes.io/projected/6c9068e7-5aea-467f-ad19-b6421524c6de-kube-api-access-ddr9r\") pod \"kuadrant-console-plugin-7bbc6bb9d7-tvp9w\" (UID: \"6c9068e7-5aea-467f-ad19-b6421524c6de\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-tvp9w"
Oct 30 12:40:25.832599 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:25.832573    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c9068e7-5aea-467f-ad19-b6421524c6de-plugin-serving-cert\") pod \"kuadrant-console-plugin-7bbc6bb9d7-tvp9w\" (UID: \"6c9068e7-5aea-467f-ad19-b6421524c6de\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-tvp9w"
Oct 30 12:40:25.854964 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:25.854937    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddr9r\" (UniqueName: \"kubernetes.io/projected/6c9068e7-5aea-467f-ad19-b6421524c6de-kube-api-access-ddr9r\") pod \"kuadrant-console-plugin-7bbc6bb9d7-tvp9w\" (UID: \"6c9068e7-5aea-467f-ad19-b6421524c6de\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-tvp9w"
Oct 30 12:40:26.011781 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:26.011750    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-tvp9w"
Oct 30 12:40:26.301491 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:26.301456    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-tvp9w"]
Oct 30 12:40:26.303769 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:40:26.303739    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c9068e7_5aea_467f_ad19_b6421524c6de.slice/crio-83bcebd5677028ca4aa857ed3bb651b487738db257c29f8f0949ec0f20a8889a WatchSource:0}: Error finding container 83bcebd5677028ca4aa857ed3bb651b487738db257c29f8f0949ec0f20a8889a: Status 404 returned error can't find the container with id 83bcebd5677028ca4aa857ed3bb651b487738db257c29f8f0949ec0f20a8889a
Oct 30 12:40:26.585991 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:26.585905    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-tvp9w" event={"ID":"6c9068e7-5aea-467f-ad19-b6421524c6de","Type":"ContainerStarted","Data":"83bcebd5677028ca4aa857ed3bb651b487738db257c29f8f0949ec0f20a8889a"}
Oct 30 12:40:29.603488 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:29.603447    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-2nng6" event={"ID":"17d543a6-06a9-46ac-9b53-e43b0c0b34f5","Type":"ContainerStarted","Data":"4bc1d1de049614c50fb78111be94d052f8f037cac20dd2e9b40f502bec9d36de"}
Oct 30 12:40:29.603960 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:29.603515    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-2nng6"
Oct 30 12:40:29.603960 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:29.603878    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-tvp9w" event={"ID":"6c9068e7-5aea-467f-ad19-b6421524c6de","Type":"ContainerStarted","Data":"550fbbe8b4868b2dd54838d2dbdd0b1c3847ca995d14261196a74fec481a47fc"}
Oct 30 12:40:29.643059 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:29.643002    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-2nng6" podStartSLOduration=2.432278826 podStartE2EDuration="5.642987243s" totalImagesPullingTime="3.210708417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:29.642188294 +0000 UTC m=+218.166472515" watchObservedRunningTime="2026-10-30 12:40:29.642987243 +0000 UTC m=+218.167271462"
Oct 30 12:40:29.697416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:29.697340    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-tvp9w" podStartSLOduration=2.00400522 podStartE2EDuration="4.697290858s" totalImagesPullingTime="2.693285638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:29.69612998 +0000 UTC m=+218.220414214" watchObservedRunningTime="2026-10-30 12:40:29.697290858 +0000 UTC m=+218.221575139"
Oct 30 12:40:40.611108 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:40.611073    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-2nng6"
Oct 30 12:40:45.094516 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:40:45.094478    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:40:54.425308 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.425267    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-dc45784b6-t2t85"]
Oct 30 12:40:54.437331 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.437294    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-dc45784b6-t2t85"
Oct 30 12:40:54.459517 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.459490    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 12:40:54.459754 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.459490    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 12:40:54.459754 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.459490    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 12:40:54.459894 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.459561    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 12:40:54.459894 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.459644    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jkbzr\""
Oct 30 12:40:54.460584 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.460566    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-dc45784b6-t2t85"]
Oct 30 12:40:54.597664 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.597627    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d0188ec-41eb-45b4-ae5c-705341675edf-tmp\") pod \"kube-auth-proxy-dc45784b6-t2t85\" (UID: \"5d0188ec-41eb-45b4-ae5c-705341675edf\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-t2t85"
Oct 30 12:40:54.597840 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.597748    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgv9s\" (UniqueName: \"kubernetes.io/projected/5d0188ec-41eb-45b4-ae5c-705341675edf-kube-api-access-xgv9s\") pod \"kube-auth-proxy-dc45784b6-t2t85\" (UID: \"5d0188ec-41eb-45b4-ae5c-705341675edf\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-t2t85"
Oct 30 12:40:54.597840 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.597805    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d0188ec-41eb-45b4-ae5c-705341675edf-tls-certs\") pod \"kube-auth-proxy-dc45784b6-t2t85\" (UID: \"5d0188ec-41eb-45b4-ae5c-705341675edf\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-t2t85"
Oct 30 12:40:54.698832 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.698749    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgv9s\" (UniqueName: \"kubernetes.io/projected/5d0188ec-41eb-45b4-ae5c-705341675edf-kube-api-access-xgv9s\") pod \"kube-auth-proxy-dc45784b6-t2t85\" (UID: \"5d0188ec-41eb-45b4-ae5c-705341675edf\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-t2t85"
Oct 30 12:40:54.698832 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.698814    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d0188ec-41eb-45b4-ae5c-705341675edf-tls-certs\") pod \"kube-auth-proxy-dc45784b6-t2t85\" (UID: \"5d0188ec-41eb-45b4-ae5c-705341675edf\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-t2t85"
Oct 30 12:40:54.699051 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.698851    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d0188ec-41eb-45b4-ae5c-705341675edf-tmp\") pod \"kube-auth-proxy-dc45784b6-t2t85\" (UID: \"5d0188ec-41eb-45b4-ae5c-705341675edf\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-t2t85"
Oct 30 12:40:54.701429 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.701382    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d0188ec-41eb-45b4-ae5c-705341675edf-tmp\") pod \"kube-auth-proxy-dc45784b6-t2t85\" (UID: \"5d0188ec-41eb-45b4-ae5c-705341675edf\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-t2t85"
Oct 30 12:40:54.701639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.701620    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d0188ec-41eb-45b4-ae5c-705341675edf-tls-certs\") pod \"kube-auth-proxy-dc45784b6-t2t85\" (UID: \"5d0188ec-41eb-45b4-ae5c-705341675edf\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-t2t85"
Oct 30 12:40:54.724834 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.724800    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgv9s\" (UniqueName: \"kubernetes.io/projected/5d0188ec-41eb-45b4-ae5c-705341675edf-kube-api-access-xgv9s\") pod \"kube-auth-proxy-dc45784b6-t2t85\" (UID: \"5d0188ec-41eb-45b4-ae5c-705341675edf\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-t2t85"
Oct 30 12:40:54.750509 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:54.750471    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-dc45784b6-t2t85"
Oct 30 12:40:55.047728 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:55.047693    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-dc45784b6-t2t85"]
Oct 30 12:40:55.053441 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:40:55.053388    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0188ec_41eb_45b4_ae5c_705341675edf.slice/crio-371de40562ff24f574202d55aed70771ae0d0e57767a6f82634507b5514137d0 WatchSource:0}: Error finding container 371de40562ff24f574202d55aed70771ae0d0e57767a6f82634507b5514137d0: Status 404 returned error can't find the container with id 371de40562ff24f574202d55aed70771ae0d0e57767a6f82634507b5514137d0
Oct 30 12:40:55.715561 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:55.715511    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-dc45784b6-t2t85" event={"ID":"5d0188ec-41eb-45b4-ae5c-705341675edf","Type":"ContainerStarted","Data":"371de40562ff24f574202d55aed70771ae0d0e57767a6f82634507b5514137d0"}
Oct 30 12:40:55.774570 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:55.774532    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74ccc46849-fttc6"]
Oct 30 12:40:55.777614 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:55.777588    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ccc46849-fttc6"
Oct 30 12:40:55.792588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:55.792546    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-snqng\""
Oct 30 12:40:55.805030 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:55.804995    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74ccc46849-fttc6"]
Oct 30 12:40:55.910691 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:55.910660    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxstp\" (UniqueName: \"kubernetes.io/projected/062ae7f7-f697-4ee4-852a-559e314fd08a-kube-api-access-wxstp\") pod \"authorino-74ccc46849-fttc6\" (UID: \"062ae7f7-f697-4ee4-852a-559e314fd08a\") " pod="kuadrant-system/authorino-74ccc46849-fttc6"
Oct 30 12:40:56.011500 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:56.011457    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxstp\" (UniqueName: \"kubernetes.io/projected/062ae7f7-f697-4ee4-852a-559e314fd08a-kube-api-access-wxstp\") pod \"authorino-74ccc46849-fttc6\" (UID: \"062ae7f7-f697-4ee4-852a-559e314fd08a\") " pod="kuadrant-system/authorino-74ccc46849-fttc6"
Oct 30 12:40:56.030879 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:56.030840    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxstp\" (UniqueName: \"kubernetes.io/projected/062ae7f7-f697-4ee4-852a-559e314fd08a-kube-api-access-wxstp\") pod \"authorino-74ccc46849-fttc6\" (UID: \"062ae7f7-f697-4ee4-852a-559e314fd08a\") " pod="kuadrant-system/authorino-74ccc46849-fttc6"
Oct 30 12:40:56.095911 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:56.095574    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ccc46849-fttc6"
Oct 30 12:40:56.403666 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:56.403578    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74ccc46849-fttc6"]
Oct 30 12:40:56.407064 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:40:56.407029    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod062ae7f7_f697_4ee4_852a_559e314fd08a.slice/crio-9cc666d3526984f20b9a18e05c740cbb401b77452f21cc43952fdaaa6a5610cb WatchSource:0}: Error finding container 9cc666d3526984f20b9a18e05c740cbb401b77452f21cc43952fdaaa6a5610cb: Status 404 returned error can't find the container with id 9cc666d3526984f20b9a18e05c740cbb401b77452f21cc43952fdaaa6a5610cb
Oct 30 12:40:56.723442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:56.723337    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ccc46849-fttc6" event={"ID":"062ae7f7-f697-4ee4-852a-559e314fd08a","Type":"ContainerStarted","Data":"9cc666d3526984f20b9a18e05c740cbb401b77452f21cc43952fdaaa6a5610cb"}
Oct 30 12:40:57.185441 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.185375    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"]
Oct 30 12:40:57.189900 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.189874    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:40:57.207588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.207546    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"]
Oct 30 12:40:57.217725 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.217693    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 12:40:57.217725 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.217726    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 12:40:57.218006 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.217837    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-t54cl\""
Oct 30 12:40:57.218780 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.218756    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 12:40:57.324278 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.324237    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bc99439-435d-4db5-b1b1-5fbe500043f7-cert\") pod \"jobset-controller-manager-6c944f4b6d-zl7n2\" (UID: \"1bc99439-435d-4db5-b1b1-5fbe500043f7\") " pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:40:57.324489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.324286    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1bc99439-435d-4db5-b1b1-5fbe500043f7-manager-config\") pod \"jobset-controller-manager-6c944f4b6d-zl7n2\" (UID: \"1bc99439-435d-4db5-b1b1-5fbe500043f7\") " pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:40:57.324489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.324312    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc99439-435d-4db5-b1b1-5fbe500043f7-metrics-certs\") pod \"jobset-controller-manager-6c944f4b6d-zl7n2\" (UID: \"1bc99439-435d-4db5-b1b1-5fbe500043f7\") " pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:40:57.324489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.324486    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldb6c\" (UniqueName: \"kubernetes.io/projected/1bc99439-435d-4db5-b1b1-5fbe500043f7-kube-api-access-ldb6c\") pod \"jobset-controller-manager-6c944f4b6d-zl7n2\" (UID: \"1bc99439-435d-4db5-b1b1-5fbe500043f7\") " pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:40:57.425683 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.425648    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bc99439-435d-4db5-b1b1-5fbe500043f7-cert\") pod \"jobset-controller-manager-6c944f4b6d-zl7n2\" (UID: \"1bc99439-435d-4db5-b1b1-5fbe500043f7\") " pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:40:57.425807 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.425701    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1bc99439-435d-4db5-b1b1-5fbe500043f7-manager-config\") pod \"jobset-controller-manager-6c944f4b6d-zl7n2\" (UID: \"1bc99439-435d-4db5-b1b1-5fbe500043f7\") " pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:40:57.425807 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.425728    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc99439-435d-4db5-b1b1-5fbe500043f7-metrics-certs\") pod \"jobset-controller-manager-6c944f4b6d-zl7n2\" (UID: \"1bc99439-435d-4db5-b1b1-5fbe500043f7\") " pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:40:57.425807 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.425791    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldb6c\" (UniqueName: \"kubernetes.io/projected/1bc99439-435d-4db5-b1b1-5fbe500043f7-kube-api-access-ldb6c\") pod \"jobset-controller-manager-6c944f4b6d-zl7n2\" (UID: \"1bc99439-435d-4db5-b1b1-5fbe500043f7\") " pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:40:57.427137 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.427092    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1bc99439-435d-4db5-b1b1-5fbe500043f7-manager-config\") pod \"jobset-controller-manager-6c944f4b6d-zl7n2\" (UID: \"1bc99439-435d-4db5-b1b1-5fbe500043f7\") " pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:40:57.429189 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.429162    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bc99439-435d-4db5-b1b1-5fbe500043f7-cert\") pod \"jobset-controller-manager-6c944f4b6d-zl7n2\" (UID: \"1bc99439-435d-4db5-b1b1-5fbe500043f7\") " pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:40:57.429282 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.429227    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc99439-435d-4db5-b1b1-5fbe500043f7-metrics-certs\") pod \"jobset-controller-manager-6c944f4b6d-zl7n2\" (UID: \"1bc99439-435d-4db5-b1b1-5fbe500043f7\") " pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:40:57.444861 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.444796    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldb6c\" (UniqueName: \"kubernetes.io/projected/1bc99439-435d-4db5-b1b1-5fbe500043f7-kube-api-access-ldb6c\") pod \"jobset-controller-manager-6c944f4b6d-zl7n2\" (UID: \"1bc99439-435d-4db5-b1b1-5fbe500043f7\") " pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:40:57.507188 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.507152    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:40:57.728699 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.728599    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-dc45784b6-t2t85" event={"ID":"5d0188ec-41eb-45b4-ae5c-705341675edf","Type":"ContainerStarted","Data":"16e02e72ef05b75f44858240538feec6632efba84c9a7b6189f5edf9eab70ee3"}
Oct 30 12:40:57.766021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.765961    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-dc45784b6-t2t85" podStartSLOduration=1.411200864 podStartE2EDuration="3.765945119s" totalImagesPullingTime="2.354744255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:57.76500867 +0000 UTC m=+246.289292913" watchObservedRunningTime="2026-10-30 12:40:57.765945119 +0000 UTC m=+246.290229339"
Oct 30 12:40:57.803829 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:57.803792    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"]
Oct 30 12:40:57.806477 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:40:57.806448    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc99439_435d_4db5_b1b1_5fbe500043f7.slice/crio-c1fcc60b05788ca0b9c7e0221009007ca8ef9346496c0ffd16ee07b7888db43b WatchSource:0}: Error finding container c1fcc60b05788ca0b9c7e0221009007ca8ef9346496c0ffd16ee07b7888db43b: Status 404 returned error can't find the container with id c1fcc60b05788ca0b9c7e0221009007ca8ef9346496c0ffd16ee07b7888db43b
Oct 30 12:40:58.734069 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:40:58.734034    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2" event={"ID":"1bc99439-435d-4db5-b1b1-5fbe500043f7","Type":"ContainerStarted","Data":"c1fcc60b05788ca0b9c7e0221009007ca8ef9346496c0ffd16ee07b7888db43b"}
Oct 30 12:41:01.751044 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:01.751005    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2" event={"ID":"1bc99439-435d-4db5-b1b1-5fbe500043f7","Type":"ContainerStarted","Data":"2c735e9b5d09a64316f44059f957d88a2686b28f9f353a89d32df5d46bd45b36"}
Oct 30 12:41:01.751498 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:01.751193    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:41:01.800785 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:01.800727    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2" podStartSLOduration=1.6762741920000002 podStartE2EDuration="4.800708677s" totalImagesPullingTime="3.124434485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:01.798884402 +0000 UTC m=+250.323168621" watchObservedRunningTime="2026-10-30 12:41:01.800708677 +0000 UTC m=+250.324992894"
Oct 30 12:41:06.572756 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:06.572695    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74ccc46849-fttc6"]
Oct 30 12:41:07.780333 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:07.780298    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ccc46849-fttc6" event={"ID":"062ae7f7-f697-4ee4-852a-559e314fd08a","Type":"ContainerStarted","Data":"3a21f386bf617965faced45741d586b3d0efbaf62bb31b5b53920e02587b3a70"}
Oct 30 12:41:07.780876 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:07.780330    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-74ccc46849-fttc6" podUID="062ae7f7-f697-4ee4-852a-559e314fd08a" containerName="authorino" containerID="cri-o://3a21f386bf617965faced45741d586b3d0efbaf62bb31b5b53920e02587b3a70" gracePeriod=30
Oct 30 12:41:07.817073 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:07.817023    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-74ccc46849-fttc6" podStartSLOduration=2.0913038999999998 podStartE2EDuration="12.817006127s" totalImagesPullingTime="10.725702227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:07.815814125 +0000 UTC m=+256.340098343" watchObservedRunningTime="2026-10-30 12:41:07.817006127 +0000 UTC m=+256.341290339"
Oct 30 12:41:08.029696 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:08.029667    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ccc46849-fttc6"
Oct 30 12:41:08.127377 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:08.127282    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/062ae7f7-f697-4ee4-852a-559e314fd08a-kube-api-access-wxstp\" (UniqueName: \"kubernetes.io/projected/062ae7f7-f697-4ee4-852a-559e314fd08a-kube-api-access-wxstp\") pod \"062ae7f7-f697-4ee4-852a-559e314fd08a\" (UID: \"062ae7f7-f697-4ee4-852a-559e314fd08a\") "
Oct 30 12:41:08.129824 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:08.129795    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062ae7f7-f697-4ee4-852a-559e314fd08a-kube-api-access-wxstp" pod "062ae7f7-f697-4ee4-852a-559e314fd08a" (UID: "062ae7f7-f697-4ee4-852a-559e314fd08a"). InnerVolumeSpecName "kube-api-access-wxstp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:41:08.228969 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:08.228934    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxstp\" (UniqueName: \"kubernetes.io/projected/062ae7f7-f697-4ee4-852a-559e314fd08a-kube-api-access-wxstp\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:41:08.785566 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:08.785532    5837 generic.go:356] "Generic (PLEG): container finished" podID="062ae7f7-f697-4ee4-852a-559e314fd08a" containerID="3a21f386bf617965faced45741d586b3d0efbaf62bb31b5b53920e02587b3a70" exitCode=0
Oct 30 12:41:08.786119 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:08.785573    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ccc46849-fttc6"
Oct 30 12:41:08.786119 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:08.785618    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ccc46849-fttc6" event={"ID":"062ae7f7-f697-4ee4-852a-559e314fd08a","Type":"ContainerDied","Data":"3a21f386bf617965faced45741d586b3d0efbaf62bb31b5b53920e02587b3a70"}
Oct 30 12:41:08.786119 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:08.785665    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ccc46849-fttc6" event={"ID":"062ae7f7-f697-4ee4-852a-559e314fd08a","Type":"ContainerDied","Data":"9cc666d3526984f20b9a18e05c740cbb401b77452f21cc43952fdaaa6a5610cb"}
Oct 30 12:41:08.786119 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:08.785694    5837 scope.go:122] "RemoveContainer" containerID="3a21f386bf617965faced45741d586b3d0efbaf62bb31b5b53920e02587b3a70"
Oct 30 12:41:08.794553 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:08.794535    5837 scope.go:122] "RemoveContainer" containerID="3a21f386bf617965faced45741d586b3d0efbaf62bb31b5b53920e02587b3a70"
Oct 30 12:41:08.794850 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:41:08.794830    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a21f386bf617965faced45741d586b3d0efbaf62bb31b5b53920e02587b3a70\": container with ID starting with 3a21f386bf617965faced45741d586b3d0efbaf62bb31b5b53920e02587b3a70 not found: ID does not exist" containerID="3a21f386bf617965faced45741d586b3d0efbaf62bb31b5b53920e02587b3a70"
Oct 30 12:41:08.794908 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:08.794861    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a21f386bf617965faced45741d586b3d0efbaf62bb31b5b53920e02587b3a70"} err="failed to get container status \"3a21f386bf617965faced45741d586b3d0efbaf62bb31b5b53920e02587b3a70\": rpc error: code = NotFound desc = could not find container \"3a21f386bf617965faced45741d586b3d0efbaf62bb31b5b53920e02587b3a70\": container with ID starting with 3a21f386bf617965faced45741d586b3d0efbaf62bb31b5b53920e02587b3a70 not found: ID does not exist"
Oct 30 12:41:08.837460 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:08.837431    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74ccc46849-fttc6"]
Oct 30 12:41:08.842316 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:08.842289    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-74ccc46849-fttc6"]
Oct 30 12:41:09.603204 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:09.603170    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="062ae7f7-f697-4ee4-852a-559e314fd08a" path="/var/lib/kubelet/pods/062ae7f7-f697-4ee4-852a-559e314fd08a/volumes"
Oct 30 12:41:12.760638 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:12.760607    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6c944f4b6d-zl7n2"
Oct 30 12:41:15.094243 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:41:15.094202    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:41:45.094747 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:41:45.094711    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:41:51.633640 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:51.633609    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 12:41:51.634072 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:41:51.633761    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 12:42:15.094561 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:42:15.094521    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:42:33.345304 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.345264    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7597569d74-g74pm"]
Oct 30 12:42:33.345832 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.345810    5837 cpu_manager.go:414] "containerMap: removing container" podUID="062ae7f7-f697-4ee4-852a-559e314fd08a" containerName="authorino"
Oct 30 12:42:33.345898 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.345835    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="062ae7f7-f697-4ee4-852a-559e314fd08a" containerName="authorino"
Oct 30 12:42:33.346056 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.346042    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="062ae7f7-f697-4ee4-852a-559e314fd08a" containerName="authorino"
Oct 30 12:42:33.348978 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.348957    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7597569d74-g74pm"
Oct 30 12:42:33.364192 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.364167    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7597569d74-g74pm"]
Oct 30 12:42:33.374778 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.374754    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-snqng\""
Oct 30 12:42:33.426308 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.426272    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmxtv\" (UniqueName: \"kubernetes.io/projected/a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a-kube-api-access-bmxtv\") pod \"authorino-7597569d74-g74pm\" (UID: \"a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a\") " pod="kuadrant-system/authorino-7597569d74-g74pm"
Oct 30 12:42:33.526783 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.526748    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxtv\" (UniqueName: \"kubernetes.io/projected/a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a-kube-api-access-bmxtv\") pod \"authorino-7597569d74-g74pm\" (UID: \"a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a\") " pod="kuadrant-system/authorino-7597569d74-g74pm"
Oct 30 12:42:33.529699 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.529671    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7597569d74-g74pm"]
Oct 30 12:42:33.530016 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:42:33.529992    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-bmxtv], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7597569d74-g74pm" podUID="a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a"
Oct 30 12:42:33.546547 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.546517    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmxtv\" (UniqueName: \"kubernetes.io/projected/a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a-kube-api-access-bmxtv\") pod \"authorino-7597569d74-g74pm\" (UID: \"a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a\") " pod="kuadrant-system/authorino-7597569d74-g74pm"
Oct 30 12:42:33.784387 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.784353    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74db894f99-ks6w4"]
Oct 30 12:42:33.788019 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.787997    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74db894f99-ks6w4"
Oct 30 12:42:33.807150 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.807124    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74db894f99-ks6w4"]
Oct 30 12:42:33.814636 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.814614    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 12:42:33.830442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.830415    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d70531a7-4794-4f11-ac8a-80c56f50dfa1-tls-cert\") pod \"authorino-74db894f99-ks6w4\" (UID: \"d70531a7-4794-4f11-ac8a-80c56f50dfa1\") " pod="kuadrant-system/authorino-74db894f99-ks6w4"
Oct 30 12:42:33.830614 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.830544    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt7z6\" (UniqueName: \"kubernetes.io/projected/d70531a7-4794-4f11-ac8a-80c56f50dfa1-kube-api-access-zt7z6\") pod \"authorino-74db894f99-ks6w4\" (UID: \"d70531a7-4794-4f11-ac8a-80c56f50dfa1\") " pod="kuadrant-system/authorino-74db894f99-ks6w4"
Oct 30 12:42:33.931942 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.931898    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d70531a7-4794-4f11-ac8a-80c56f50dfa1-tls-cert\") pod \"authorino-74db894f99-ks6w4\" (UID: \"d70531a7-4794-4f11-ac8a-80c56f50dfa1\") " pod="kuadrant-system/authorino-74db894f99-ks6w4"
Oct 30 12:42:33.932119 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.932011    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt7z6\" (UniqueName: \"kubernetes.io/projected/d70531a7-4794-4f11-ac8a-80c56f50dfa1-kube-api-access-zt7z6\") pod \"authorino-74db894f99-ks6w4\" (UID: \"d70531a7-4794-4f11-ac8a-80c56f50dfa1\") " pod="kuadrant-system/authorino-74db894f99-ks6w4"
Oct 30 12:42:33.934738 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.934711    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d70531a7-4794-4f11-ac8a-80c56f50dfa1-tls-cert\") pod \"authorino-74db894f99-ks6w4\" (UID: \"d70531a7-4794-4f11-ac8a-80c56f50dfa1\") " pod="kuadrant-system/authorino-74db894f99-ks6w4"
Oct 30 12:42:33.951253 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:33.951214    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt7z6\" (UniqueName: \"kubernetes.io/projected/d70531a7-4794-4f11-ac8a-80c56f50dfa1-kube-api-access-zt7z6\") pod \"authorino-74db894f99-ks6w4\" (UID: \"d70531a7-4794-4f11-ac8a-80c56f50dfa1\") " pod="kuadrant-system/authorino-74db894f99-ks6w4"
Oct 30 12:42:34.099954 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:34.099851    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74db894f99-ks6w4"
Oct 30 12:42:34.163684 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:34.163654    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7597569d74-g74pm"
Oct 30 12:42:34.173582 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:34.173553    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7597569d74-g74pm"
Oct 30 12:42:34.234230 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:34.234198    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a-kube-api-access-bmxtv\" (UniqueName: \"kubernetes.io/projected/a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a-kube-api-access-bmxtv\") pod \"a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a\" (UID: \"a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a\") "
Oct 30 12:42:34.236571 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:34.236539    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a-kube-api-access-bmxtv" pod "a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a" (UID: "a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a"). InnerVolumeSpecName "kube-api-access-bmxtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:42:34.335487 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:34.335445    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmxtv\" (UniqueName: \"kubernetes.io/projected/a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a-kube-api-access-bmxtv\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:42:34.391343 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:34.391259    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74db894f99-ks6w4"]
Oct 30 12:42:34.395272 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:42:34.395245    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd70531a7_4794_4f11_ac8a_80c56f50dfa1.slice/crio-a56a82638f553ada5566e53c3f79efea8edce667b91d18e028d7b75e7fd172ab WatchSource:0}: Error finding container a56a82638f553ada5566e53c3f79efea8edce667b91d18e028d7b75e7fd172ab: Status 404 returned error can't find the container with id a56a82638f553ada5566e53c3f79efea8edce667b91d18e028d7b75e7fd172ab
Oct 30 12:42:34.396823 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:34.396805    5837 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:42:34.537197 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:34.537163    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74db894f99-ks6w4"]
Oct 30 12:42:35.169707 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:35.169678    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7597569d74-g74pm"
Oct 30 12:42:35.169707 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:35.169701    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74db894f99-ks6w4" event={"ID":"d70531a7-4794-4f11-ac8a-80c56f50dfa1","Type":"ContainerStarted","Data":"a56a82638f553ada5566e53c3f79efea8edce667b91d18e028d7b75e7fd172ab"}
Oct 30 12:42:35.246138 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:35.245683    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7597569d74-g74pm"]
Oct 30 12:42:35.247282 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:35.247250    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7597569d74-g74pm"]
Oct 30 12:42:35.603316 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:35.603275    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a" path="/var/lib/kubelet/pods/a0a34604-ba9d-41a4-8e5d-cbf0eeb0020a/volumes"
Oct 30 12:42:36.177047 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:36.177009    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74db894f99-ks6w4" event={"ID":"d70531a7-4794-4f11-ac8a-80c56f50dfa1","Type":"ContainerStarted","Data":"8170e37aa515e441b966e8ac5e8503fd18d3081d2db4fc24b986137e7870a1b1"}
Oct 30 12:42:36.177254 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:36.177055    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-74db894f99-ks6w4" podUID="d70531a7-4794-4f11-ac8a-80c56f50dfa1" containerName="authorino" containerID="cri-o://8170e37aa515e441b966e8ac5e8503fd18d3081d2db4fc24b986137e7870a1b1" gracePeriod=30
Oct 30 12:42:36.215646 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:36.215589    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-74db894f99-ks6w4" podStartSLOduration=2.409913582 podStartE2EDuration="3.215574021s" totalImagesPullingTime="805.660439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:36.214629745 +0000 UTC m=+344.738913964" watchObservedRunningTime="2026-10-30 12:42:36.215574021 +0000 UTC m=+344.739858220"
Oct 30 12:42:36.426628 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:36.426602    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74db894f99-ks6w4"
Oct 30 12:42:36.454362 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:36.454264    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d70531a7-4794-4f11-ac8a-80c56f50dfa1-tls-cert\" (UniqueName: \"kubernetes.io/secret/d70531a7-4794-4f11-ac8a-80c56f50dfa1-tls-cert\") pod \"d70531a7-4794-4f11-ac8a-80c56f50dfa1\" (UID: \"d70531a7-4794-4f11-ac8a-80c56f50dfa1\") "
Oct 30 12:42:36.454362 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:36.454327    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70531a7-4794-4f11-ac8a-80c56f50dfa1-kube-api-access-zt7z6\" (UniqueName: \"kubernetes.io/projected/d70531a7-4794-4f11-ac8a-80c56f50dfa1-kube-api-access-zt7z6\") pod \"d70531a7-4794-4f11-ac8a-80c56f50dfa1\" (UID: \"d70531a7-4794-4f11-ac8a-80c56f50dfa1\") "
Oct 30 12:42:36.456537 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:36.456504    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70531a7-4794-4f11-ac8a-80c56f50dfa1-kube-api-access-zt7z6" pod "d70531a7-4794-4f11-ac8a-80c56f50dfa1" (UID: "d70531a7-4794-4f11-ac8a-80c56f50dfa1"). InnerVolumeSpecName "kube-api-access-zt7z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:42:36.465763 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:36.465729    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70531a7-4794-4f11-ac8a-80c56f50dfa1-tls-cert" pod "d70531a7-4794-4f11-ac8a-80c56f50dfa1" (UID: "d70531a7-4794-4f11-ac8a-80c56f50dfa1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:42:36.555920 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:36.555882    5837 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d70531a7-4794-4f11-ac8a-80c56f50dfa1-tls-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:42:36.555920 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:36.555916    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt7z6\" (UniqueName: \"kubernetes.io/projected/d70531a7-4794-4f11-ac8a-80c56f50dfa1-kube-api-access-zt7z6\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:42:37.182095 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:37.182061    5837 generic.go:356] "Generic (PLEG): container finished" podID="d70531a7-4794-4f11-ac8a-80c56f50dfa1" containerID="8170e37aa515e441b966e8ac5e8503fd18d3081d2db4fc24b986137e7870a1b1" exitCode=0
Oct 30 12:42:37.182542 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:37.182113    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74db894f99-ks6w4"
Oct 30 12:42:37.182542 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:37.182158    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74db894f99-ks6w4" event={"ID":"d70531a7-4794-4f11-ac8a-80c56f50dfa1","Type":"ContainerDied","Data":"8170e37aa515e441b966e8ac5e8503fd18d3081d2db4fc24b986137e7870a1b1"}
Oct 30 12:42:37.182542 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:37.182201    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74db894f99-ks6w4" event={"ID":"d70531a7-4794-4f11-ac8a-80c56f50dfa1","Type":"ContainerDied","Data":"a56a82638f553ada5566e53c3f79efea8edce667b91d18e028d7b75e7fd172ab"}
Oct 30 12:42:37.182542 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:37.182220    5837 scope.go:122] "RemoveContainer" containerID="8170e37aa515e441b966e8ac5e8503fd18d3081d2db4fc24b986137e7870a1b1"
Oct 30 12:42:37.191602 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:37.191581    5837 scope.go:122] "RemoveContainer" containerID="8170e37aa515e441b966e8ac5e8503fd18d3081d2db4fc24b986137e7870a1b1"
Oct 30 12:42:37.191910 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:42:37.191888    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8170e37aa515e441b966e8ac5e8503fd18d3081d2db4fc24b986137e7870a1b1\": container with ID starting with 8170e37aa515e441b966e8ac5e8503fd18d3081d2db4fc24b986137e7870a1b1 not found: ID does not exist" containerID="8170e37aa515e441b966e8ac5e8503fd18d3081d2db4fc24b986137e7870a1b1"
Oct 30 12:42:37.191961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:37.191924    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8170e37aa515e441b966e8ac5e8503fd18d3081d2db4fc24b986137e7870a1b1"} err="failed to get container status \"8170e37aa515e441b966e8ac5e8503fd18d3081d2db4fc24b986137e7870a1b1\": rpc error: code = NotFound desc = could not find container \"8170e37aa515e441b966e8ac5e8503fd18d3081d2db4fc24b986137e7870a1b1\": container with ID starting with 8170e37aa515e441b966e8ac5e8503fd18d3081d2db4fc24b986137e7870a1b1 not found: ID does not exist"
Oct 30 12:42:37.229004 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:37.228921    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74db894f99-ks6w4"]
Oct 30 12:42:37.232975 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:37.232951    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-74db894f99-ks6w4"]
Oct 30 12:42:37.603764 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:42:37.603731    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70531a7-4794-4f11-ac8a-80c56f50dfa1" path="/var/lib/kubelet/pods/d70531a7-4794-4f11-ac8a-80c56f50dfa1/volumes"
Oct 30 12:42:45.093780 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:42:45.093743    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:43:15.094247 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:43:15.094163    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:43:45.094700 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:43:45.094657    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:44:15.093901 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:44:15.093860    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:44:34.154636 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.154554    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"]
Oct 30 12:44:34.155071 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.154942    5837 cpu_manager.go:414] "containerMap: removing container" podUID="d70531a7-4794-4f11-ac8a-80c56f50dfa1" containerName="authorino"
Oct 30 12:44:34.155071 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.154952    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70531a7-4794-4f11-ac8a-80c56f50dfa1" containerName="authorino"
Oct 30 12:44:34.155071 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.155023    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70531a7-4794-4f11-ac8a-80c56f50dfa1" containerName="authorino"
Oct 30 12:44:34.158195 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.158175    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.175367 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.175341    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 12:44:34.186647 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.186592    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 12:44:34.186647 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.186592    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 12:44:34.186647 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.186626    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-76488\""
Oct 30 12:44:34.188242 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.188217    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"]
Oct 30 12:44:34.227226 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.227180    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.227439 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.227299    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.227439 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.227334    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-489pd\" (UniqueName: \"kubernetes.io/projected/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-kube-api-access-489pd\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.227439 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.227378    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.227558 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.227493    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.227597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.227576    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.328981 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.328941    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.329136 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.328998    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.329136 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.329025    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.329136 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.329043    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-489pd\" (UniqueName: \"kubernetes.io/projected/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-kube-api-access-489pd\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.329136 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.329086    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.329136 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.329133    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.329543 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.329514    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.329682 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.329563    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.329682 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.329598    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.331498 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.331479    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.331883 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.331862    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.350175 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.350140    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-489pd\" (UniqueName: \"kubernetes.io/projected/20d9a8aa-0955-49d8-bb05-d1a6fa52f560-kube-api-access-489pd\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pdnxs\" (UID: \"20d9a8aa-0955-49d8-bb05-d1a6fa52f560\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.473275 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.473185    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:34.768500 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:34.768269    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"]
Oct 30 12:44:34.771786 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:44:34.771761    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d9a8aa_0955_49d8_bb05_d1a6fa52f560.slice/crio-06429b203b3b10e3801754dd9beffeb699d496baf5e6dad1ccffeed08ea2b53e WatchSource:0}: Error finding container 06429b203b3b10e3801754dd9beffeb699d496baf5e6dad1ccffeed08ea2b53e: Status 404 returned error can't find the container with id 06429b203b3b10e3801754dd9beffeb699d496baf5e6dad1ccffeed08ea2b53e
Oct 30 12:44:35.707084 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:35.707042    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs" event={"ID":"20d9a8aa-0955-49d8-bb05-d1a6fa52f560","Type":"ContainerStarted","Data":"06429b203b3b10e3801754dd9beffeb699d496baf5e6dad1ccffeed08ea2b53e"}
Oct 30 12:44:36.712998 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:36.712959    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs" event={"ID":"20d9a8aa-0955-49d8-bb05-d1a6fa52f560","Type":"ContainerStarted","Data":"f48cd53cb1c7e35569d080ec2f63306cf9428ac14a7d254f380ef714c3775c03"}
Oct 30 12:44:36.752765 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:36.752704    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs" podStartSLOduration=1.731137848 podStartE2EDuration="2.752686102s" totalImagesPullingTime="1.021548254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:36.75224547 +0000 UTC m=+465.276529688" watchObservedRunningTime="2026-10-30 12:44:36.752686102 +0000 UTC m=+465.276970322"
Oct 30 12:44:37.151947 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.151914    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"]
Oct 30 12:44:37.155981 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.155960    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.173656 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.173626    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 12:44:37.185190 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.185166    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"]
Oct 30 12:44:37.257057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.256998    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c06bb10a-6170-4c87-82e8-4695cdddc789-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.257057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.257059    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xpg5\" (UniqueName: \"kubernetes.io/projected/c06bb10a-6170-4c87-82e8-4695cdddc789-kube-api-access-8xpg5\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.257282 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.257181    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c06bb10a-6170-4c87-82e8-4695cdddc789-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.257282 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.257247    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c06bb10a-6170-4c87-82e8-4695cdddc789-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.257362 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.257305    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c06bb10a-6170-4c87-82e8-4695cdddc789-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.257362 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.257341    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c06bb10a-6170-4c87-82e8-4695cdddc789-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.358761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.358720    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c06bb10a-6170-4c87-82e8-4695cdddc789-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.358975 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.358783    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c06bb10a-6170-4c87-82e8-4695cdddc789-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.358975 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.358887    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xpg5\" (UniqueName: \"kubernetes.io/projected/c06bb10a-6170-4c87-82e8-4695cdddc789-kube-api-access-8xpg5\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.359094 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.358982    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c06bb10a-6170-4c87-82e8-4695cdddc789-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.359094 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.359052    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c06bb10a-6170-4c87-82e8-4695cdddc789-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.359094 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.359078    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c06bb10a-6170-4c87-82e8-4695cdddc789-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.359244 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.359125    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c06bb10a-6170-4c87-82e8-4695cdddc789-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.359335 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.359314    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c06bb10a-6170-4c87-82e8-4695cdddc789-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.359381 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.359367    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c06bb10a-6170-4c87-82e8-4695cdddc789-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.361294 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.361272    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c06bb10a-6170-4c87-82e8-4695cdddc789-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.361603 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.361586    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c06bb10a-6170-4c87-82e8-4695cdddc789-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.378096 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.378068    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xpg5\" (UniqueName: \"kubernetes.io/projected/c06bb10a-6170-4c87-82e8-4695cdddc789-kube-api-access-8xpg5\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-vhr7q\" (UID: \"c06bb10a-6170-4c87-82e8-4695cdddc789\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.469195 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.469102    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:37.473461 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.473387    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:37.765805 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:37.765778    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"]
Oct 30 12:44:37.772807 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:44:37.772776    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06bb10a_6170_4c87_82e8_4695cdddc789.slice/crio-071f767cb5ec42b77b7591ed3e608069dd1551581ecff0502938e0fcfe950606 WatchSource:0}: Error finding container 071f767cb5ec42b77b7591ed3e608069dd1551581ecff0502938e0fcfe950606: Status 404 returned error can't find the container with id 071f767cb5ec42b77b7591ed3e608069dd1551581ecff0502938e0fcfe950606
Oct 30 12:44:38.723926 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:38.723886    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q" event={"ID":"c06bb10a-6170-4c87-82e8-4695cdddc789","Type":"ContainerStarted","Data":"ce51c15739fd193c643d604692259eee832dd68b1dad60f0972f75503784fd89"}
Oct 30 12:44:38.724126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:38.723957    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q" event={"ID":"c06bb10a-6170-4c87-82e8-4695cdddc789","Type":"ContainerStarted","Data":"071f767cb5ec42b77b7591ed3e608069dd1551581ecff0502938e0fcfe950606"}
Oct 30 12:44:38.766040 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:38.765969    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q" podStartSLOduration=1.583755868 podStartE2EDuration="1.765954264s" totalImagesPullingTime="182.198396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:38.76531897 +0000 UTC m=+467.289603188" watchObservedRunningTime="2026-10-30 12:44:38.765954264 +0000 UTC m=+467.290238531"
Oct 30 12:44:39.470241 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.470205    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:39.782251 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.782188    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-8wtq4"]
Oct 30 12:44:39.786750 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.786727    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.803983 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.803923    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 12:44:39.817222 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.817190    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-8wtq4"]
Oct 30 12:44:39.883226 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.883192    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f38956-a205-4c3a-ab46-aa50d995cc2d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.883451 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.883336    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f38956-a205-4c3a-ab46-aa50d995cc2d-home\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.883451 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.883444    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f38956-a205-4c3a-ab46-aa50d995cc2d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.883588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.883493    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f38956-a205-4c3a-ab46-aa50d995cc2d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.883588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.883511    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f38956-a205-4c3a-ab46-aa50d995cc2d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.883718 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.883690    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqkmg\" (UniqueName: \"kubernetes.io/projected/80f38956-a205-4c3a-ab46-aa50d995cc2d-kube-api-access-zqkmg\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.984688 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.984654    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f38956-a205-4c3a-ab46-aa50d995cc2d-home\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.984688 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.984687    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f38956-a205-4c3a-ab46-aa50d995cc2d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.984943 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.984724    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f38956-a205-4c3a-ab46-aa50d995cc2d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.984943 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.984741    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f38956-a205-4c3a-ab46-aa50d995cc2d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.984943 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.984789    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqkmg\" (UniqueName: \"kubernetes.io/projected/80f38956-a205-4c3a-ab46-aa50d995cc2d-kube-api-access-zqkmg\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.984943 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.984826    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f38956-a205-4c3a-ab46-aa50d995cc2d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.985142 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.985121    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f38956-a205-4c3a-ab46-aa50d995cc2d-home\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.985203 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.985183    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f38956-a205-4c3a-ab46-aa50d995cc2d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.985249 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.985226    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f38956-a205-4c3a-ab46-aa50d995cc2d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.987305 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.987281    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f38956-a205-4c3a-ab46-aa50d995cc2d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:39.987587 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:39.987567    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f38956-a205-4c3a-ab46-aa50d995cc2d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:40.004216 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:40.004183    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqkmg\" (UniqueName: \"kubernetes.io/projected/80f38956-a205-4c3a-ab46-aa50d995cc2d-kube-api-access-zqkmg\") pod \"echo-simulator-kserve-64b6f8f96c-8wtq4\" (UID: \"80f38956-a205-4c3a-ab46-aa50d995cc2d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:40.109427 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:40.109321    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:40.405448 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:40.405417    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-8wtq4"]
Oct 30 12:44:40.407604 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:44:40.407574    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f38956_a205_4c3a_ab46_aa50d995cc2d.slice/crio-ba061d50b9225c6995b100b0476d322e22011aafecfe0ad57f453567408d975d WatchSource:0}: Error finding container ba061d50b9225c6995b100b0476d322e22011aafecfe0ad57f453567408d975d: Status 404 returned error can't find the container with id ba061d50b9225c6995b100b0476d322e22011aafecfe0ad57f453567408d975d
Oct 30 12:44:40.734808 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:40.734772    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4" event={"ID":"80f38956-a205-4c3a-ab46-aa50d995cc2d","Type":"ContainerStarted","Data":"ba061d50b9225c6995b100b0476d322e22011aafecfe0ad57f453567408d975d"}
Oct 30 12:44:42.746787 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:42.746746    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4" event={"ID":"80f38956-a205-4c3a-ab46-aa50d995cc2d","Type":"ContainerStarted","Data":"9627d6628b83563acdd033bba4149ada242a5f16ee7d087af723a4fa9eaab6be"}
Oct 30 12:44:42.786589 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:42.786525    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4" podStartSLOduration=2.294477353 podStartE2EDuration="3.786505126s" totalImagesPullingTime="1.492027773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:42.786027684 +0000 UTC m=+471.310311902" watchObservedRunningTime="2026-10-30 12:44:42.786505126 +0000 UTC m=+471.310789347"
Oct 30 12:44:42.849236 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:42.849189    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"]
Oct 30 12:44:42.853186 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:42.853164    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:42.870440 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:42.870417    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 12:44:42.877299 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:42.877272    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"]
Oct 30 12:44:43.017293 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.017204    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b175788a-b7a3-4aab-a17f-1362ac6b83b5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.017293 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.017253    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b175788a-b7a3-4aab-a17f-1362ac6b83b5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.017521 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.017368    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b175788a-b7a3-4aab-a17f-1362ac6b83b5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.017521 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.017442    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b175788a-b7a3-4aab-a17f-1362ac6b83b5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.017521 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.017463    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdbrv\" (UniqueName: \"kubernetes.io/projected/b175788a-b7a3-4aab-a17f-1362ac6b83b5-kube-api-access-gdbrv\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.017521 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.017499    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b175788a-b7a3-4aab-a17f-1362ac6b83b5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.110214 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.110160    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:43.118338 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.118308    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b175788a-b7a3-4aab-a17f-1362ac6b83b5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.118529 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.118356    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b175788a-b7a3-4aab-a17f-1362ac6b83b5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.118529 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.118446    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b175788a-b7a3-4aab-a17f-1362ac6b83b5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.118529 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.118507    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b175788a-b7a3-4aab-a17f-1362ac6b83b5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.118648 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.118622    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdbrv\" (UniqueName: \"kubernetes.io/projected/b175788a-b7a3-4aab-a17f-1362ac6b83b5-kube-api-access-gdbrv\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.118691 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.118676    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b175788a-b7a3-4aab-a17f-1362ac6b83b5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.118814 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.118790    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b175788a-b7a3-4aab-a17f-1362ac6b83b5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.118932 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.118867    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b175788a-b7a3-4aab-a17f-1362ac6b83b5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.119045 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.119026    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b175788a-b7a3-4aab-a17f-1362ac6b83b5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.120979 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.120956    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b175788a-b7a3-4aab-a17f-1362ac6b83b5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.121275 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.121260    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b175788a-b7a3-4aab-a17f-1362ac6b83b5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.140128 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.140097    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdbrv\" (UniqueName: \"kubernetes.io/projected/b175788a-b7a3-4aab-a17f-1362ac6b83b5-kube-api-access-gdbrv\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq\" (UID: \"b175788a-b7a3-4aab-a17f-1362ac6b83b5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.167022 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.166976    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:43.460508 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.460459    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"]
Oct 30 12:44:43.463340 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:44:43.463309    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb175788a_b7a3_4aab_a17f_1362ac6b83b5.slice/crio-e06e26e785459e534552f3c436adc6f52f53ecbbe59e6fb6a9cf6b47b2f58c6b WatchSource:0}: Error finding container e06e26e785459e534552f3c436adc6f52f53ecbbe59e6fb6a9cf6b47b2f58c6b: Status 404 returned error can't find the container with id e06e26e785459e534552f3c436adc6f52f53ecbbe59e6fb6a9cf6b47b2f58c6b
Oct 30 12:44:43.752762 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.752713    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq" event={"ID":"b175788a-b7a3-4aab-a17f-1362ac6b83b5","Type":"ContainerStarted","Data":"a466ee40fa3e24d28967436b0da1be836b1f4d23fa095a5dc37cced5d94d2155"}
Oct 30 12:44:43.753274 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.752790    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq" event={"ID":"b175788a-b7a3-4aab-a17f-1362ac6b83b5","Type":"ContainerStarted","Data":"e06e26e785459e534552f3c436adc6f52f53ecbbe59e6fb6a9cf6b47b2f58c6b"}
Oct 30 12:44:43.808356 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:43.808294    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq" podStartSLOduration=1.636977583 podStartE2EDuration="1.808279928s" totalImagesPullingTime="171.302345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:43.807278184 +0000 UTC m=+472.331562402" watchObservedRunningTime="2026-10-30 12:44:43.808279928 +0000 UTC m=+472.332564146"
Oct 30 12:44:44.167155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:44.167070    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:44.474221 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:44.474124    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:44.486869 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:44.486839    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:44.769656 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:44.769627    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pdnxs"
Oct 30 12:44:45.094516 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:44:45.094430    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:44:47.469485 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:47.469443    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:47.482222 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:47.482187    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:47.785151 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:47.785125    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q"
Oct 30 12:44:50.110122 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:50.110071    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:50.122634 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:50.122607    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:50.799579 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:50.799551    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-8wtq4"
Oct 30 12:44:53.167864 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:53.167822    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:53.180451 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:53.180424    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:53.814543 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:53.814514    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq"
Oct 30 12:44:58.861915 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:58.861884    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"]
Oct 30 12:44:58.865437 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:58.865419    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:58.888333 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:58.888284    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 12:44:58.902059 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:58.902024    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"]
Oct 30 12:44:58.961145 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:58.961111    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bn28m\""
Oct 30 12:44:58.966550 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:58.966528    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:58.966639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:58.966560    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6d77\" (UniqueName: \"kubernetes.io/projected/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-kube-api-access-g6d77\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:58.966639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:58.966591    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:58.966746 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:58.966716    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:58.966791 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:58.966770    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:58.966840 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:58.966806    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:58.966922 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:58.966903    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:58.966962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:58.966934    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:58.966962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:58.966957    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.067873 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.067830    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.068050 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.067893    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.068050 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.067909    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.068050 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.067929    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.068050 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.067948    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.068050 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.067982    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.068300 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.068068    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.068300 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.068095    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6d77\" (UniqueName: \"kubernetes.io/projected/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-kube-api-access-g6d77\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.068449 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.068376    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.068514 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.068481    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.068571 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.068524    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.068637 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.068616    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.068681 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.068654    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.068879 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.068860    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.070659 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.070630    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.070801 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.070782    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.087047 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.087022    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6d77\" (UniqueName: \"kubernetes.io/projected/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-kube-api-access-g6d77\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.087155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.087027    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99edc7cd-e5d4-4ac9-ab07-c0b91926a654-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xhs8n\" (UID: \"99edc7cd-e5d4-4ac9-ab07-c0b91926a654\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.179636 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.179539    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:44:59.488111 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.488079    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"]
Oct 30 12:44:59.490436 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:44:59.490407    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99edc7cd_e5d4_4ac9_ab07_c0b91926a654.slice/crio-d0c90dfebd4f5997b36ccce7ff79f31927ddfd50b513a2b223ac1070370ba285 WatchSource:0}: Error finding container d0c90dfebd4f5997b36ccce7ff79f31927ddfd50b513a2b223ac1070370ba285: Status 404 returned error can't find the container with id d0c90dfebd4f5997b36ccce7ff79f31927ddfd50b513a2b223ac1070370ba285
Oct 30 12:44:59.831976 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:44:59.831888    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n" event={"ID":"99edc7cd-e5d4-4ac9-ab07-c0b91926a654","Type":"ContainerStarted","Data":"d0c90dfebd4f5997b36ccce7ff79f31927ddfd50b513a2b223ac1070370ba285"}
Oct 30 12:45:01.341092 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:01.341059    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65c6d4fc99-x6wdt"]
Oct 30 12:45:01.348423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:01.348363    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65c6d4fc99-x6wdt"
Oct 30 12:45:01.377502 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:01.377472    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 12:45:01.377671 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:01.377605    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-snqng\""
Oct 30 12:45:01.379633 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:01.379603    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65c6d4fc99-x6wdt"]
Oct 30 12:45:01.393834 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:01.393796    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4412a6ae-aaea-4aba-97f8-17ee44ff4dde-tls-cert\") pod \"authorino-65c6d4fc99-x6wdt\" (UID: \"4412a6ae-aaea-4aba-97f8-17ee44ff4dde\") " pod="kuadrant-system/authorino-65c6d4fc99-x6wdt"
Oct 30 12:45:01.394016 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:01.393882    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pjgz\" (UniqueName: \"kubernetes.io/projected/4412a6ae-aaea-4aba-97f8-17ee44ff4dde-kube-api-access-2pjgz\") pod \"authorino-65c6d4fc99-x6wdt\" (UID: \"4412a6ae-aaea-4aba-97f8-17ee44ff4dde\") " pod="kuadrant-system/authorino-65c6d4fc99-x6wdt"
Oct 30 12:45:01.495317 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:01.495282    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pjgz\" (UniqueName: \"kubernetes.io/projected/4412a6ae-aaea-4aba-97f8-17ee44ff4dde-kube-api-access-2pjgz\") pod \"authorino-65c6d4fc99-x6wdt\" (UID: \"4412a6ae-aaea-4aba-97f8-17ee44ff4dde\") " pod="kuadrant-system/authorino-65c6d4fc99-x6wdt"
Oct 30 12:45:01.495515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:01.495461    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4412a6ae-aaea-4aba-97f8-17ee44ff4dde-tls-cert\") pod \"authorino-65c6d4fc99-x6wdt\" (UID: \"4412a6ae-aaea-4aba-97f8-17ee44ff4dde\") " pod="kuadrant-system/authorino-65c6d4fc99-x6wdt"
Oct 30 12:45:01.498368 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:01.498338    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4412a6ae-aaea-4aba-97f8-17ee44ff4dde-tls-cert\") pod \"authorino-65c6d4fc99-x6wdt\" (UID: \"4412a6ae-aaea-4aba-97f8-17ee44ff4dde\") " pod="kuadrant-system/authorino-65c6d4fc99-x6wdt"
Oct 30 12:45:01.514998 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:01.514960    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pjgz\" (UniqueName: \"kubernetes.io/projected/4412a6ae-aaea-4aba-97f8-17ee44ff4dde-kube-api-access-2pjgz\") pod \"authorino-65c6d4fc99-x6wdt\" (UID: \"4412a6ae-aaea-4aba-97f8-17ee44ff4dde\") " pod="kuadrant-system/authorino-65c6d4fc99-x6wdt"
Oct 30 12:45:01.663115 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:01.663024    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65c6d4fc99-x6wdt"
Oct 30 12:45:01.962951 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:01.962913    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65c6d4fc99-x6wdt"]
Oct 30 12:45:01.965744 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:45:01.965711    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4412a6ae_aaea_4aba_97f8_17ee44ff4dde.slice/crio-46c6e922e6aba82cf9a770f00bf25a754d1c1be439a3407c873da05c3143763e WatchSource:0}: Error finding container 46c6e922e6aba82cf9a770f00bf25a754d1c1be439a3407c873da05c3143763e: Status 404 returned error can't find the container with id 46c6e922e6aba82cf9a770f00bf25a754d1c1be439a3407c873da05c3143763e
Oct 30 12:45:02.084761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:02.084729    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:45:02.084868 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:02.084801    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:45:02.084868 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:02.084832    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:45:02.850241 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:02.850208    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n" event={"ID":"99edc7cd-e5d4-4ac9-ab07-c0b91926a654","Type":"ContainerStarted","Data":"7c56ac8cc5a93b5e958e026f11acf826362bcbe5b8dfac28f52886513d11816d"}
Oct 30 12:45:02.851444 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:02.851420    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65c6d4fc99-x6wdt" event={"ID":"4412a6ae-aaea-4aba-97f8-17ee44ff4dde","Type":"ContainerStarted","Data":"46c6e922e6aba82cf9a770f00bf25a754d1c1be439a3407c873da05c3143763e"}
Oct 30 12:45:02.908360 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:02.908239    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n" podStartSLOduration=2.316951023 podStartE2EDuration="4.908221385s" totalImagesPullingTime="2.591270362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:02.90599602 +0000 UTC m=+491.430280255" watchObservedRunningTime="2026-10-30 12:45:02.908221385 +0000 UTC m=+491.432505605"
Oct 30 12:45:03.181238 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:03.181142    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:45:03.182226 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:03.182193    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xhs8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:45:03.182379 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:03.182256    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n" podUID="99edc7cd-e5d4-4ac9-ab07-c0b91926a654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 12:45:03.859083 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:03.859040    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65c6d4fc99-x6wdt" event={"ID":"4412a6ae-aaea-4aba-97f8-17ee44ff4dde","Type":"ContainerStarted","Data":"64cec709a8796c5fcd35121d93cc0e45b3a8a0329d6c5f275c831a16bcdff2d9"}
Oct 30 12:45:03.910928 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:03.908663    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-65c6d4fc99-x6wdt" podStartSLOduration=2.064388488 podStartE2EDuration="2.908641056s" totalImagesPullingTime="844.252568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.907667885 +0000 UTC m=+492.431952104" watchObservedRunningTime="2026-10-30 12:45:03.908641056 +0000 UTC m=+492.432925287"
Oct 30 12:45:04.180062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:04.179977    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xhs8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:45:04.180062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:04.180038    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n" podUID="99edc7cd-e5d4-4ac9-ab07-c0b91926a654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 12:45:05.180299 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:05.180250    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xhs8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:45:05.180851 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:05.180321    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n" podUID="99edc7cd-e5d4-4ac9-ab07-c0b91926a654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 12:45:06.180280 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:06.180238    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xhs8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:45:06.180525 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:06.180296    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n" podUID="99edc7cd-e5d4-4ac9-ab07-c0b91926a654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 12:45:07.197423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:07.189814    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xhs8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:45:07.197423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:07.189885    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n" podUID="99edc7cd-e5d4-4ac9-ab07-c0b91926a654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 12:45:08.180779 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:08.180740    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xhs8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:45:08.180975 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:08.180804    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n" podUID="99edc7cd-e5d4-4ac9-ab07-c0b91926a654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 12:45:09.180777 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:09.180734    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xhs8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:45:09.181198 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:09.180813    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n" podUID="99edc7cd-e5d4-4ac9-ab07-c0b91926a654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 12:45:10.855619 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:10.855585    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:45:10.856215 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:10.855948    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:45:10.856988 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:10.856969    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xhs8n"
Oct 30 12:45:13.887501 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:13.887463    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"]
Oct 30 12:45:13.894122 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:13.894100    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:13.945433 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:13.945354    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"]
Oct 30 12:45:13.950521 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:13.950494    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"]
Oct 30 12:45:13.961021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:13.960995    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:13.990022 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:13.989987    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"]
Oct 30 12:45:14.041576 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.041542    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.041576 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.041582    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.041829 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.041606    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9d9d\" (UniqueName: \"kubernetes.io/projected/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-kube-api-access-x9d9d\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.041829 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.041719    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.041829 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.041766    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.041829 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.041814    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.042006 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.041843    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.042006 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.041863    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.042006 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.041927    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.042006 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.041979    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.042134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.042021    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.042134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.042038    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.042134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.042091    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.042134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.042125    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.042320 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.042186    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.042320 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.042231    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.042320 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.042294    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.042320 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.042313    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx989\" (UniqueName: \"kubernetes.io/projected/8fa68fa1-c682-4b28-ad60-7c3207842d6a-kube-api-access-bx989\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.143881 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.143784    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.143881 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.143845    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx989\" (UniqueName: \"kubernetes.io/projected/8fa68fa1-c682-4b28-ad60-7c3207842d6a-kube-api-access-bx989\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.143881 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.143876    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.144163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.143901    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.144163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.143935    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9d9d\" (UniqueName: \"kubernetes.io/projected/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-kube-api-access-x9d9d\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.144163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144083    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.144163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144145    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.144358 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144198    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.144358 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144248    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.144358 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144278    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.144358 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144309    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.144607 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144358    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.144607 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144370    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.144607 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144471    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.144607 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144504    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.144607 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144530    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.144607 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144545    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.144607 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144600    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.144967 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144617    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.144967 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144661    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.144967 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144676    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.144967 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144701    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.144967 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144709    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.144967 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144874    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.145337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.144974    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.145337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.145195    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.145337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.145213    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.145517 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.145418    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.146692 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.146663    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.147298 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.147273    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.147665 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.147643    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.147785 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.147766    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.163771 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.163748    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx989\" (UniqueName: \"kubernetes.io/projected/8fa68fa1-c682-4b28-ad60-7c3207842d6a-kube-api-access-bx989\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.166440 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.166410    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.167576 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.167528    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-v7qbw\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.167817 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.167799    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9d9d\" (UniqueName: \"kubernetes.io/projected/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-kube-api-access-x9d9d\") pod \"maas-default-gateway-openshift-default-7d9897b54c-rmq6s\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.239139 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.239092    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:14.303109 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.303063    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:14.544904 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.544878    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"]
Oct 30 12:45:14.546990 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:45:14.546954    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa68fa1_c682_4b28_ad60_7c3207842d6a.slice/crio-1d40db9e7d695e17a3bee7e6ffb2b611d4cf75a9a42496b412b2585adf90e5c2 WatchSource:0}: Error finding container 1d40db9e7d695e17a3bee7e6ffb2b611d4cf75a9a42496b412b2585adf90e5c2: Status 404 returned error can't find the container with id 1d40db9e7d695e17a3bee7e6ffb2b611d4cf75a9a42496b412b2585adf90e5c2
Oct 30 12:45:14.549998 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.549970    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:45:14.550077 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.550036    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:45:14.550077 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.550069    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:45:14.600325 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.600299    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"]
Oct 30 12:45:14.603219 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:45:14.603195    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d98825f_f663_4f10_be1a_ce6e6ecc0c3d.slice/crio-75c866bc3f582ac7e2980de4f629c1de3e29f5a1986a2546ad22adac05da9e4e WatchSource:0}: Error finding container 75c866bc3f582ac7e2980de4f629c1de3e29f5a1986a2546ad22adac05da9e4e: Status 404 returned error can't find the container with id 75c866bc3f582ac7e2980de4f629c1de3e29f5a1986a2546ad22adac05da9e4e
Oct 30 12:45:14.606367 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.606334    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:45:14.606493 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.606422    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:45:14.606493 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.606456    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:45:14.959641 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.959518    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s" event={"ID":"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d","Type":"ContainerStarted","Data":"ba556f50c847aa6ffee2143e395b5783af0d19246baae41e91c215feccc67399"}
Oct 30 12:45:14.959641 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.959600    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s" event={"ID":"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d","Type":"ContainerStarted","Data":"75c866bc3f582ac7e2980de4f629c1de3e29f5a1986a2546ad22adac05da9e4e"}
Oct 30 12:45:14.961833 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.961792    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw" event={"ID":"8fa68fa1-c682-4b28-ad60-7c3207842d6a","Type":"ContainerStarted","Data":"ef1f24ddd2b406b0be258da8ebe2d4c0c91d3610d154e1fd00fc56f523238496"}
Oct 30 12:45:14.962025 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:14.961851    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw" event={"ID":"8fa68fa1-c682-4b28-ad60-7c3207842d6a","Type":"ContainerStarted","Data":"1d40db9e7d695e17a3bee7e6ffb2b611d4cf75a9a42496b412b2585adf90e5c2"}
Oct 30 12:45:15.006838 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:15.006769    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s" podStartSLOduration=2.006743576 podStartE2EDuration="2.006743576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:15.00586894 +0000 UTC m=+503.530153159" watchObservedRunningTime="2026-10-30 12:45:15.006743576 +0000 UTC m=+503.531027796"
Oct 30 12:45:15.049795 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:15.049741    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw" podStartSLOduration=2.049727638 podStartE2EDuration="2.049727638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:15.049383247 +0000 UTC m=+503.573667490" watchObservedRunningTime="2026-10-30 12:45:15.049727638 +0000 UTC m=+503.574011856"
Oct 30 12:45:15.098910 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:45:15.098869    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:45:15.242823 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:15.242786    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:15.246993 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:15.246941    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-v7qbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 12:45:15.247333 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:15.247292    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw" podUID="8fa68fa1-c682-4b28-ad60-7c3207842d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 12:45:15.305426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:15.304806    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:15.327421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:15.314534    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-rmq6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 12:45:15.327421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:15.314599    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s" podUID="1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 12:45:16.240290 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:16.240251    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-v7qbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 12:45:16.240778 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:16.240307    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw" podUID="8fa68fa1-c682-4b28-ad60-7c3207842d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 12:45:16.307423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:16.303888    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-rmq6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 12:45:16.307423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:16.303957    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s" podUID="1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 12:45:17.240199 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:17.240165    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-v7qbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 12:45:17.240424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:17.240233    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw" podUID="8fa68fa1-c682-4b28-ad60-7c3207842d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 12:45:17.304893 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:17.304845    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-rmq6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 12:45:17.305104 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:17.304917    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s" podUID="1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 12:45:18.239696 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:18.239658    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-v7qbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 12:45:18.239924 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:18.239722    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw" podUID="8fa68fa1-c682-4b28-ad60-7c3207842d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 12:45:18.303964 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:18.303924    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-rmq6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 12:45:18.304603 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:18.304568    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s" podUID="1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 12:45:19.239967 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:19.239928    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-v7qbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 12:45:19.240253 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:19.239984    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw" podUID="8fa68fa1-c682-4b28-ad60-7c3207842d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 12:45:19.303920 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:19.303875    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-rmq6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 12:45:19.304245 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:19.304213    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s" podUID="1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 12:45:20.239697 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:20.239653    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-v7qbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 12:45:20.239889 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:20.239713    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw" podUID="8fa68fa1-c682-4b28-ad60-7c3207842d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 12:45:20.307421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:20.307013    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-rmq6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 12:45:20.307421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:20.307076    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s" podUID="1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 12:45:21.240292 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:21.240237    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-v7qbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 12:45:21.240535 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:21.240314    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw" podUID="8fa68fa1-c682-4b28-ad60-7c3207842d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 12:45:21.304507 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:21.304457    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-rmq6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 12:45:21.304714 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:21.304527    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s" podUID="1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 12:45:22.240220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:22.240180    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-v7qbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 12:45:22.240797 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:22.240237    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw" podUID="8fa68fa1-c682-4b28-ad60-7c3207842d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 12:45:22.303719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:22.303676    5837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-rmq6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 12:45:22.303937 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:22.303746    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s" podUID="1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 12:45:23.271851 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:23.271813    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:23.272371 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:23.272234    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:23.273331 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:23.273306    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 12:45:23.374028 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:23.373997    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:23.374944 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:23.374821    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:23.376125 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:45:23.376097    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:45:45.095337 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:45:45.095224    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:46:02.214235 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.214140    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"]
Oct 30 12:46:02.219955 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.219931    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.240721 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.240685    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cdd5b3ae-gw-openshift-default-dockercfg-txtb9\""
Oct 30 12:46:02.254178 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.254147    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"]
Oct 30 12:46:02.288554 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.288521    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istiod-ca-cert\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.288720 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.288579    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-podinfo\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.288720 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.288646    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-data\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.288720 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.288680    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-token\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.288720 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.288703    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-workload-certs\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.288859 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.288839    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-workload-socket\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.288896 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.288873    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-credential-socket\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.288935 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.288921    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckfcm\" (UniqueName: \"kubernetes.io/projected/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-kube-api-access-ckfcm\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.288970 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.288959    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-envoy\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.390388 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.390350    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istiod-ca-cert\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.390589 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.390447    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-podinfo\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.390589 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.390480    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-data\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.390589 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.390519    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-token\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.390589 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.390548    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-workload-certs\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.390799 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.390594    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-workload-socket\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.390799 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.390626    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-credential-socket\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.390799 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.390746    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfcm\" (UniqueName: \"kubernetes.io/projected/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-kube-api-access-ckfcm\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.390964 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.390820    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-envoy\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.390964 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.390933    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-data\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.391078 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.390965    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-credential-socket\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.391130 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.391075    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-workload-certs\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.391196 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.391177    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istiod-ca-cert\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.391231 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.391202    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-workload-socket\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.393145 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.393122    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-envoy\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.393253 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.393209    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-podinfo\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.411111 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.411076    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-token\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.411834 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.411814    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckfcm\" (UniqueName: \"kubernetes.io/projected/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-kube-api-access-ckfcm\") pod \"e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") " pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.535552 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.535510    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:02.849831 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.849198    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"]
Oct 30 12:46:02.851876 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:02.851847    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3fdd85_e545_41c0_9bc8_24db9fb81cb4.slice/crio-cd1b67c6f12657426b67331646d1c56ae85b985c5791ba21337f66a6a4c46733 WatchSource:0}: Error finding container cd1b67c6f12657426b67331646d1c56ae85b985c5791ba21337f66a6a4c46733: Status 404 returned error can't find the container with id cd1b67c6f12657426b67331646d1c56ae85b985c5791ba21337f66a6a4c46733
Oct 30 12:46:02.854464 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.854368    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:02.854553 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.854503    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:02.854553 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:02.854534    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:03.226792 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.226699    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" event={"ID":"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4","Type":"ContainerStarted","Data":"437c97d3095c2d56da056c948fb1374790617c10936928207b5807b1946bde5f"}
Oct 30 12:46:03.226792 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.226752    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" event={"ID":"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4","Type":"ContainerStarted","Data":"cd1b67c6f12657426b67331646d1c56ae85b985c5791ba21337f66a6a4c46733"}
Oct 30 12:46:03.295136 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.295077    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" podStartSLOduration=1.295058758 podStartE2EDuration="1.295058758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:03.282065003 +0000 UTC m=+551.806349240" watchObservedRunningTime="2026-10-30 12:46:03.295058758 +0000 UTC m=+551.819342979"
Oct 30 12:46:03.361838 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.361799    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"]
Oct 30 12:46:03.366012 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.365990    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.387175 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.387130    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"]
Oct 30 12:46:03.394274 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.394243    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5431fa34-openshift-default-dockercfg-n2cn9\""
Oct 30 12:46:03.403180 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.403133    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-workload-socket\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.403386 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.403223    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-istio-envoy\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.403386 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.403332    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-workload-certs\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.403558 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.403408    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42e36f7-20c2-4259-99c0-a221e144d330-istio-token\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.403558 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.403519    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42e36f7-20c2-4259-99c0-a221e144d330-istio-podinfo\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.403669 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.403574    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-credential-socket\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.403669 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.403623    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btv5p\" (UniqueName: \"kubernetes.io/projected/e42e36f7-20c2-4259-99c0-a221e144d330-kube-api-access-btv5p\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.403779 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.403731    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42e36f7-20c2-4259-99c0-a221e144d330-istiod-ca-cert\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.403779 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.403767    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-istio-data\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.505323 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.505285    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42e36f7-20c2-4259-99c0-a221e144d330-istio-podinfo\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.505585 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.505334    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-credential-socket\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.505585 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.505352    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btv5p\" (UniqueName: \"kubernetes.io/projected/e42e36f7-20c2-4259-99c0-a221e144d330-kube-api-access-btv5p\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.505585 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.505418    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42e36f7-20c2-4259-99c0-a221e144d330-istiod-ca-cert\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.505585 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.505455    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-istio-data\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.505585 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.505505    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-workload-socket\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.505585 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.505569    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-istio-envoy\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.505904 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.505611    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-workload-certs\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.505904 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.505668    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42e36f7-20c2-4259-99c0-a221e144d330-istio-token\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.506004 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.505939    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-credential-socket\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.506057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.506001    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-workload-socket\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.506092 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.506062    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-workload-certs\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.506135 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.506091    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-istio-data\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.506461 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.506439    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42e36f7-20c2-4259-99c0-a221e144d330-istiod-ca-cert\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.508053 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.508030    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-istio-envoy\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.508282 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.508264    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42e36f7-20c2-4259-99c0-a221e144d330-istio-podinfo\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.524681 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.524650    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42e36f7-20c2-4259-99c0-a221e144d330-istio-token\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.524792 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.524701    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btv5p\" (UniqueName: \"kubernetes.io/projected/e42e36f7-20c2-4259-99c0-a221e144d330-kube-api-access-btv5p\") pod \"e2e-ait-5431fa34-openshift-default-6685769f9-p95b7\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") " pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.536631 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.536607    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:03.541951 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.541926    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:03.681737 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.681701    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:03.989070 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.989027    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"]
Oct 30 12:46:03.990737 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:03.990706    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42e36f7_20c2_4259_99c0_a221e144d330.slice/crio-fdd786d48c7a7d99a79fc6fa7d6f7033190afc26eeb07c1b61520306eac71f07 WatchSource:0}: Error finding container fdd786d48c7a7d99a79fc6fa7d6f7033190afc26eeb07c1b61520306eac71f07: Status 404 returned error can't find the container with id fdd786d48c7a7d99a79fc6fa7d6f7033190afc26eeb07c1b61520306eac71f07
Oct 30 12:46:03.993551 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.993521    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:03.993634 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.993586    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:03.993634 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:03.993620    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:04.236085 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:04.236037    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7" event={"ID":"e42e36f7-20c2-4259-99c0-a221e144d330","Type":"ContainerStarted","Data":"df466c4ae60f81ed662c1b8738a26830e68a8702e3e6ba31b330ed4df0b255b9"}
Oct 30 12:46:04.236589 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:04.236107    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7" event={"ID":"e42e36f7-20c2-4259-99c0-a221e144d330","Type":"ContainerStarted","Data":"fdd786d48c7a7d99a79fc6fa7d6f7033190afc26eeb07c1b61520306eac71f07"}
Oct 30 12:46:04.236589 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:04.236139    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:04.237459 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:04.237375    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:04.279719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:04.279653    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7" podStartSLOduration=1.279631895 podStartE2EDuration="1.279631895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:04.276575406 +0000 UTC m=+552.800859652" watchObservedRunningTime="2026-10-30 12:46:04.279631895 +0000 UTC m=+552.803916115"
Oct 30 12:46:04.681923 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:04.681826    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:05.687381 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:05.687350    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:06.248923 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:06.248884    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:06.249769 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:06.249749    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:08.115724 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.115688    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"]
Oct 30 12:46:08.119766 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.119736    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"
Oct 30 12:46:08.143824 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.143792    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"]
Oct 30 12:46:08.147650 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.147626    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4da58cb4\""
Oct 30 12:46:08.157285 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.157243    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834dc833-1f8e-439c-a15d-09bf0f851fe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q\" (UID: \"834dc833-1f8e-439c-a15d-09bf0f851fe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"
Oct 30 12:46:08.157548 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.157527    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdr8s\" (UniqueName: \"kubernetes.io/projected/834dc833-1f8e-439c-a15d-09bf0f851fe7-kube-api-access-zdr8s\") pod \"payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q\" (UID: \"834dc833-1f8e-439c-a15d-09bf0f851fe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"
Oct 30 12:46:08.206091 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.206063    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4da58cb4-dockercfg-pgmlf\""
Oct 30 12:46:08.258261 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.258228    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdr8s\" (UniqueName: \"kubernetes.io/projected/834dc833-1f8e-439c-a15d-09bf0f851fe7-kube-api-access-zdr8s\") pod \"payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q\" (UID: \"834dc833-1f8e-439c-a15d-09bf0f851fe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"
Oct 30 12:46:08.258453 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.258281    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834dc833-1f8e-439c-a15d-09bf0f851fe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q\" (UID: \"834dc833-1f8e-439c-a15d-09bf0f851fe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"
Oct 30 12:46:08.259022 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.259002    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834dc833-1f8e-439c-a15d-09bf0f851fe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q\" (UID: \"834dc833-1f8e-439c-a15d-09bf0f851fe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"
Oct 30 12:46:08.277737 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.277710    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdr8s\" (UniqueName: \"kubernetes.io/projected/834dc833-1f8e-439c-a15d-09bf0f851fe7-kube-api-access-zdr8s\") pod \"payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q\" (UID: \"834dc833-1f8e-439c-a15d-09bf0f851fe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"
Oct 30 12:46:08.437052 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.436962    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"
Oct 30 12:46:08.732161 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.732031    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"]
Oct 30 12:46:08.735092 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:08.735047    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834dc833_1f8e_439c_a15d_09bf0f851fe7.slice/crio-8fbc6fd46364c3d496b484cce115445af4e6a484e6123822247e504dcec33753 WatchSource:0}: Error finding container 8fbc6fd46364c3d496b484cce115445af4e6a484e6123822247e504dcec33753: Status 404 returned error can't find the container with id 8fbc6fd46364c3d496b484cce115445af4e6a484e6123822247e504dcec33753
Oct 30 12:46:08.979762 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.979730    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"]
Oct 30 12:46:08.984432 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.984347    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"
Oct 30 12:46:08.988957 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.988927    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"]
Oct 30 12:46:08.992878 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:08.992847    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"
Oct 30 12:46:09.011711 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.011507    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"]
Oct 30 12:46:09.012866 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.012841    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c1b2ed0a\""
Oct 30 12:46:09.013341 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.013319    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c1b2ed0a-dockercfg-784d9\""
Oct 30 12:46:09.016271 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.016244    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"]
Oct 30 12:46:09.069839 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.069804    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc4p9\" (UniqueName: \"kubernetes.io/projected/478ecd8b-4281-4ec1-9aca-bfd2284a435e-kube-api-access-hc4p9\") pod \"payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s\" (UID: \"478ecd8b-4281-4ec1-9aca-bfd2284a435e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"
Oct 30 12:46:09.070020 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.069934    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9dc715-8ac8-483f-ba94-5427d2e6ed2b-plugins-config-volume\") pod \"payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg\" (UID: \"6a9dc715-8ac8-483f-ba94-5427d2e6ed2b\") " pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"
Oct 30 12:46:09.070020 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.069973    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92fsb\" (UniqueName: \"kubernetes.io/projected/6a9dc715-8ac8-483f-ba94-5427d2e6ed2b-kube-api-access-92fsb\") pod \"payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg\" (UID: \"6a9dc715-8ac8-483f-ba94-5427d2e6ed2b\") " pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"
Oct 30 12:46:09.070109 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.070064    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478ecd8b-4281-4ec1-9aca-bfd2284a435e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s\" (UID: \"478ecd8b-4281-4ec1-9aca-bfd2284a435e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"
Oct 30 12:46:09.172366 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.171696    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9dc715-8ac8-483f-ba94-5427d2e6ed2b-plugins-config-volume\") pod \"payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg\" (UID: \"6a9dc715-8ac8-483f-ba94-5427d2e6ed2b\") " pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"
Oct 30 12:46:09.172366 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.171844    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92fsb\" (UniqueName: \"kubernetes.io/projected/6a9dc715-8ac8-483f-ba94-5427d2e6ed2b-kube-api-access-92fsb\") pod \"payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg\" (UID: \"6a9dc715-8ac8-483f-ba94-5427d2e6ed2b\") " pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"
Oct 30 12:46:09.172366 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.171919    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478ecd8b-4281-4ec1-9aca-bfd2284a435e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s\" (UID: \"478ecd8b-4281-4ec1-9aca-bfd2284a435e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"
Oct 30 12:46:09.172366 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.172008    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4p9\" (UniqueName: \"kubernetes.io/projected/478ecd8b-4281-4ec1-9aca-bfd2284a435e-kube-api-access-hc4p9\") pod \"payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s\" (UID: \"478ecd8b-4281-4ec1-9aca-bfd2284a435e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"
Oct 30 12:46:09.173090 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.172478    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9dc715-8ac8-483f-ba94-5427d2e6ed2b-plugins-config-volume\") pod \"payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg\" (UID: \"6a9dc715-8ac8-483f-ba94-5427d2e6ed2b\") " pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"
Oct 30 12:46:09.173090 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.172883    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478ecd8b-4281-4ec1-9aca-bfd2284a435e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s\" (UID: \"478ecd8b-4281-4ec1-9aca-bfd2284a435e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"
Oct 30 12:46:09.197115 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.197079    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc4p9\" (UniqueName: \"kubernetes.io/projected/478ecd8b-4281-4ec1-9aca-bfd2284a435e-kube-api-access-hc4p9\") pod \"payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s\" (UID: \"478ecd8b-4281-4ec1-9aca-bfd2284a435e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"
Oct 30 12:46:09.197282 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.197259    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92fsb\" (UniqueName: \"kubernetes.io/projected/6a9dc715-8ac8-483f-ba94-5427d2e6ed2b-kube-api-access-92fsb\") pod \"payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg\" (UID: \"6a9dc715-8ac8-483f-ba94-5427d2e6ed2b\") " pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"
Oct 30 12:46:09.270670 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.270569    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q" event={"ID":"834dc833-1f8e-439c-a15d-09bf0f851fe7","Type":"ContainerStarted","Data":"8fbc6fd46364c3d496b484cce115445af4e6a484e6123822247e504dcec33753"}
Oct 30 12:46:09.299205 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.299162    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"
Oct 30 12:46:09.317603 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.317558    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"
Oct 30 12:46:09.617627 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.617232    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"]
Oct 30 12:46:09.619336 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:09.619306    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478ecd8b_4281_4ec1_9aca_bfd2284a435e.slice/crio-6df56c3c192caae70c2115c60ef98f826be8641d8b4fe2284529555bd7717968 WatchSource:0}: Error finding container 6df56c3c192caae70c2115c60ef98f826be8641d8b4fe2284529555bd7717968: Status 404 returned error can't find the container with id 6df56c3c192caae70c2115c60ef98f826be8641d8b4fe2284529555bd7717968
Oct 30 12:46:09.649817 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.649783    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"]
Oct 30 12:46:09.651521 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:09.651488    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9dc715_8ac8_483f_ba94_5427d2e6ed2b.slice/crio-89f1a68052d6b4cb62018a1e13347b391cd05266678b110337b980a601a62335 WatchSource:0}: Error finding container 89f1a68052d6b4cb62018a1e13347b391cd05266678b110337b980a601a62335: Status 404 returned error can't find the container with id 89f1a68052d6b4cb62018a1e13347b391cd05266678b110337b980a601a62335
Oct 30 12:46:09.838841 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.838797    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"]
Oct 30 12:46:09.843760 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.843735    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"
Oct 30 12:46:09.859807 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.859755    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5431fa34-dockercfg-7nlbv\""
Oct 30 12:46:09.860226 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.860203    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5431fa34\""
Oct 30 12:46:09.862382 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.862355    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"]
Oct 30 12:46:09.882932 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.882843    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bfec5e9-47b4-4090-a6ef-69172ebbf69f-plugins-config-volume\") pod \"payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn\" (UID: \"5bfec5e9-47b4-4090-a6ef-69172ebbf69f\") " pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"
Oct 30 12:46:09.883103 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.882931    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbpw\" (UniqueName: \"kubernetes.io/projected/5bfec5e9-47b4-4090-a6ef-69172ebbf69f-kube-api-access-8bbpw\") pod \"payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn\" (UID: \"5bfec5e9-47b4-4090-a6ef-69172ebbf69f\") " pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"
Oct 30 12:46:09.911157 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.911123    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"]
Oct 30 12:46:09.915978 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.915945    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"
Oct 30 12:46:09.937611 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.937578    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"]
Oct 30 12:46:09.985492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.984882    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111eeb2b-43df-4895-a5d5-b101432ac848-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf\" (UID: \"111eeb2b-43df-4895-a5d5-b101432ac848\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"
Oct 30 12:46:09.985492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.984960    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctnb7\" (UniqueName: \"kubernetes.io/projected/111eeb2b-43df-4895-a5d5-b101432ac848-kube-api-access-ctnb7\") pod \"payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf\" (UID: \"111eeb2b-43df-4895-a5d5-b101432ac848\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"
Oct 30 12:46:09.985492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.985113    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bfec5e9-47b4-4090-a6ef-69172ebbf69f-plugins-config-volume\") pod \"payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn\" (UID: \"5bfec5e9-47b4-4090-a6ef-69172ebbf69f\") " pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"
Oct 30 12:46:09.985492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.985165    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbpw\" (UniqueName: \"kubernetes.io/projected/5bfec5e9-47b4-4090-a6ef-69172ebbf69f-kube-api-access-8bbpw\") pod \"payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn\" (UID: \"5bfec5e9-47b4-4090-a6ef-69172ebbf69f\") " pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"
Oct 30 12:46:09.988984 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:09.988924    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bfec5e9-47b4-4090-a6ef-69172ebbf69f-plugins-config-volume\") pod \"payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn\" (UID: \"5bfec5e9-47b4-4090-a6ef-69172ebbf69f\") " pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"
Oct 30 12:46:10.008526 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.008491    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbpw\" (UniqueName: \"kubernetes.io/projected/5bfec5e9-47b4-4090-a6ef-69172ebbf69f-kube-api-access-8bbpw\") pod \"payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn\" (UID: \"5bfec5e9-47b4-4090-a6ef-69172ebbf69f\") " pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"
Oct 30 12:46:10.086141 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.086104    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111eeb2b-43df-4895-a5d5-b101432ac848-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf\" (UID: \"111eeb2b-43df-4895-a5d5-b101432ac848\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"
Oct 30 12:46:10.086343 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.086162    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctnb7\" (UniqueName: \"kubernetes.io/projected/111eeb2b-43df-4895-a5d5-b101432ac848-kube-api-access-ctnb7\") pod \"payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf\" (UID: \"111eeb2b-43df-4895-a5d5-b101432ac848\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"
Oct 30 12:46:10.087104 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.087015    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111eeb2b-43df-4895-a5d5-b101432ac848-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf\" (UID: \"111eeb2b-43df-4895-a5d5-b101432ac848\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"
Oct 30 12:46:10.110223 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.110119    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctnb7\" (UniqueName: \"kubernetes.io/projected/111eeb2b-43df-4895-a5d5-b101432ac848-kube-api-access-ctnb7\") pod \"payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf\" (UID: \"111eeb2b-43df-4895-a5d5-b101432ac848\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"
Oct 30 12:46:10.160437 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.160324    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"
Oct 30 12:46:10.195451 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.193381    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"]
Oct 30 12:46:10.199777 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.199002    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.229482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.223522    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"]
Oct 30 12:46:10.240846 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.240726    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"
Oct 30 12:46:10.262741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.262716    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-48623dd7-openshift-default-dockercfg-tjxlh\""
Oct 30 12:46:10.282268 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.282236    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg" event={"ID":"6a9dc715-8ac8-483f-ba94-5427d2e6ed2b","Type":"ContainerStarted","Data":"89f1a68052d6b4cb62018a1e13347b391cd05266678b110337b980a601a62335"}
Oct 30 12:46:10.284157 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.284131    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s" event={"ID":"478ecd8b-4281-4ec1-9aca-bfd2284a435e","Type":"ContainerStarted","Data":"6df56c3c192caae70c2115c60ef98f826be8641d8b4fe2284529555bd7717968"}
Oct 30 12:46:10.416479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.391601    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-workload-certs\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.416479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.391667    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-workload-socket\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.416479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.391700    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-data\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.416479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.391796    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-envoy\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.416479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.391838    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147b05ae-0fc7-49bd-8521-8a092279d46c-istiod-ca-cert\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.416479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.391876    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-credential-socket\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.416479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.391937    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-token\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.416479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.391979    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-podinfo\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.416479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.392031    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-485rm\" (UniqueName: \"kubernetes.io/projected/147b05ae-0fc7-49bd-8521-8a092279d46c-kube-api-access-485rm\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.493950 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.493784    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-token\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.493950 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.493846    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-podinfo\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.495572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.494499    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-485rm\" (UniqueName: \"kubernetes.io/projected/147b05ae-0fc7-49bd-8521-8a092279d46c-kube-api-access-485rm\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.495572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.494632    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-workload-certs\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.495572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.494675    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-workload-socket\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.495572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.494723    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-data\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.495572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.494812    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-envoy\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.495572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.494853    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147b05ae-0fc7-49bd-8521-8a092279d46c-istiod-ca-cert\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.495572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.494892    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-credential-socket\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.495572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.495176    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-workload-certs\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.495572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.495259    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-credential-socket\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.496084 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.495838    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-data\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.496285 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.496251    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-workload-socket\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.496426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.496362    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147b05ae-0fc7-49bd-8521-8a092279d46c-istiod-ca-cert\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.502286 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.502256    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-podinfo\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.503572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.503543    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-envoy\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.517160 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.517128    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-token\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.525100 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.524993    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"]
Oct 30 12:46:10.526537 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.526508    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-485rm\" (UniqueName: \"kubernetes.io/projected/147b05ae-0fc7-49bd-8521-8a092279d46c-kube-api-access-485rm\") pod \"e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") " pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.527729 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.527708    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:10.580995 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.580672    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"]
Oct 30 12:46:10.581160 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:10.581037    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111eeb2b_43df_4895_a5d5_b101432ac848.slice/crio-43ce0520b81686efb8240bfeb3607550f425f1c3534fff07a6e51289c5312b97 WatchSource:0}: Error finding container 43ce0520b81686efb8240bfeb3607550f425f1c3534fff07a6e51289c5312b97: Status 404 returned error can't find the container with id 43ce0520b81686efb8240bfeb3607550f425f1c3534fff07a6e51289c5312b97
Oct 30 12:46:10.874063 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.874034    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"]
Oct 30 12:46:10.878537 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:10.878484    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147b05ae_0fc7_49bd_8521_8a092279d46c.slice/crio-a47627496cbb13263279ec2b8b621a601e742f0d7e13b58ad8459466e46c4065 WatchSource:0}: Error finding container a47627496cbb13263279ec2b8b621a601e742f0d7e13b58ad8459466e46c4065: Status 404 returned error can't find the container with id a47627496cbb13263279ec2b8b621a601e742f0d7e13b58ad8459466e46c4065
Oct 30 12:46:10.883763 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.883722    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:10.883923 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.883823    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:10.883923 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:10.883868    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:11.316340 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:11.316300    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn" event={"ID":"5bfec5e9-47b4-4090-a6ef-69172ebbf69f","Type":"ContainerStarted","Data":"df47d3ae42f2a1c10bd8f699025a341f2d695a7a4bd5b1c65fa73d2505040681"}
Oct 30 12:46:11.332679 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:11.332623    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf" event={"ID":"111eeb2b-43df-4895-a5d5-b101432ac848","Type":"ContainerStarted","Data":"43ce0520b81686efb8240bfeb3607550f425f1c3534fff07a6e51289c5312b97"}
Oct 30 12:46:11.337838 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:11.337798    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd" event={"ID":"147b05ae-0fc7-49bd-8521-8a092279d46c","Type":"ContainerStarted","Data":"ef05778be2425a7cab00379c1d0898b7eb34f711a9e65a183cdc89a4479d8315"}
Oct 30 12:46:11.338025 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:11.337861    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd" event={"ID":"147b05ae-0fc7-49bd-8521-8a092279d46c","Type":"ContainerStarted","Data":"a47627496cbb13263279ec2b8b621a601e742f0d7e13b58ad8459466e46c4065"}
Oct 30 12:46:11.387057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:11.386731    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd" podStartSLOduration=1.386708507 podStartE2EDuration="1.386708507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:11.38403918 +0000 UTC m=+559.908323398" watchObservedRunningTime="2026-10-30 12:46:11.386708507 +0000 UTC m=+559.910992729"
Oct 30 12:46:11.530796 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:11.530661    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:12.539435 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:12.539363    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:13.362783 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:13.362703    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn" event={"ID":"5bfec5e9-47b4-4090-a6ef-69172ebbf69f","Type":"ContainerStarted","Data":"e23d76598b85b22b629117147edd20be3e2a803290e5c865ae14a446497aa63b"}
Oct 30 12:46:13.362968 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:13.362833    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"
Oct 30 12:46:13.365782 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:13.365753    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf" event={"ID":"111eeb2b-43df-4895-a5d5-b101432ac848","Type":"ContainerStarted","Data":"c5775f9213884e37b0b0f77e0ac7b27e03a85b83778a9092e1a1fe29bad9a578"}
Oct 30 12:46:13.365926 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:13.365862    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"
Oct 30 12:46:13.367776 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:13.367739    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q" event={"ID":"834dc833-1f8e-439c-a15d-09bf0f851fe7","Type":"ContainerStarted","Data":"0d05fb92af6cd3b46d6c7659fcfc115009ec79938f4384be353eef4a1c03f06c"}
Oct 30 12:46:13.368300 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:13.367986    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:13.368300 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:13.368041    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"
Oct 30 12:46:13.369173 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:13.369113    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:13.442588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:13.442532    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn" podStartSLOduration=1.977331574 podStartE2EDuration="4.442515123s" totalImagesPullingTime="2.465183549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:13.441326158 +0000 UTC m=+561.965610378" watchObservedRunningTime="2026-10-30 12:46:13.442515123 +0000 UTC m=+561.966799342"
Oct 30 12:46:13.511458 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:13.503646    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q" podStartSLOduration=1.764023683 podStartE2EDuration="5.503625274s" totalImagesPullingTime="3.739601591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:13.50094459 +0000 UTC m=+562.025228811" watchObservedRunningTime="2026-10-30 12:46:13.503625274 +0000 UTC m=+562.027909510"
Oct 30 12:46:14.375058 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:14.375018    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg" event={"ID":"6a9dc715-8ac8-483f-ba94-5427d2e6ed2b","Type":"ContainerStarted","Data":"d4180c6701c974f4c313d137606b658e62a472e391af248080154bd6c8a6e612"}
Oct 30 12:46:14.375570 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:14.375081    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"
Oct 30 12:46:14.376644 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:14.376591    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s" event={"ID":"478ecd8b-4281-4ec1-9aca-bfd2284a435e","Type":"ContainerStarted","Data":"0c9722f2bb7f5fb33d691e8e1e5a06b53d4be5671f01a15cd6cf94e1d391a32f"}
Oct 30 12:46:14.431070 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:14.431015    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf" podStartSLOduration=2.97591942 podStartE2EDuration="5.430996168s" totalImagesPullingTime="2.455076748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:13.599359905 +0000 UTC m=+562.123644124" watchObservedRunningTime="2026-10-30 12:46:14.430996168 +0000 UTC m=+562.955280388"
Oct 30 12:46:14.432164 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:14.432121    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg" podStartSLOduration=2.32723374 podStartE2EDuration="6.432106701s" totalImagesPullingTime="4.104872961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:14.429239185 +0000 UTC m=+562.953523404" watchObservedRunningTime="2026-10-30 12:46:14.432106701 +0000 UTC m=+562.956390920"
Oct 30 12:46:14.474343 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:14.474256    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s" podStartSLOduration=2.327558573 podStartE2EDuration="6.474224707s" totalImagesPullingTime="4.146666134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:14.472238632 +0000 UTC m=+562.996522869" watchObservedRunningTime="2026-10-30 12:46:14.474224707 +0000 UTC m=+562.998508927"
Oct 30 12:46:15.117879 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:46:15.117828    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:46:15.318818 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:15.318773    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"]
Oct 30 12:46:15.324499 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:15.324021    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"
Oct 30 12:46:15.341892 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:15.341758    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-48623dd7\""
Oct 30 12:46:15.341892 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:15.341799    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-48623dd7-dockercfg-dx66b\""
Oct 30 12:46:15.346553 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:15.345832    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"]
Oct 30 12:46:15.383412 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:15.383304    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"
Oct 30 12:46:15.418073 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:15.418042    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s78ww\" (UniqueName: \"kubernetes.io/projected/1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d-kube-api-access-s78ww\") pod \"payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm\" (UID: \"1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d\") " pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"
Oct 30 12:46:15.418277 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:15.418099    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d-plugins-config-volume\") pod \"payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm\" (UID: \"1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d\") " pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"
Oct 30 12:46:15.519718 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:15.519679    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78ww\" (UniqueName: \"kubernetes.io/projected/1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d-kube-api-access-s78ww\") pod \"payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm\" (UID: \"1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d\") " pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"
Oct 30 12:46:15.519885 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:15.519730    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d-plugins-config-volume\") pod \"payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm\" (UID: \"1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d\") " pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"
Oct 30 12:46:15.520360 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:15.520338    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d-plugins-config-volume\") pod \"payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm\" (UID: \"1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d\") " pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"
Oct 30 12:46:15.539947 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:15.539919    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78ww\" (UniqueName: \"kubernetes.io/projected/1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d-kube-api-access-s78ww\") pod \"payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm\" (UID: \"1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d\") " pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"
Oct 30 12:46:15.647486 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:15.647368    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"
Oct 30 12:46:15.964752 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:15.964722    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"]
Oct 30 12:46:15.968785 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:15.968755    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7adf61_0dc7_4e45_ae9e_86f6cd1e8f9d.slice/crio-3ec80b07b27501f75418d529b7b053923ce19497bc4e1199871994375ea1d8ba WatchSource:0}: Error finding container 3ec80b07b27501f75418d529b7b053923ce19497bc4e1199871994375ea1d8ba: Status 404 returned error can't find the container with id 3ec80b07b27501f75418d529b7b053923ce19497bc4e1199871994375ea1d8ba
Oct 30 12:46:16.412595 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:16.412558    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm" event={"ID":"1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d","Type":"ContainerStarted","Data":"c84bc09853ccb112268ceb1a9ac22c3aac04a7ecfabeeea41655fb83bcd1ebf4"}
Oct 30 12:46:16.413046 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:16.412622    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm" event={"ID":"1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d","Type":"ContainerStarted","Data":"3ec80b07b27501f75418d529b7b053923ce19497bc4e1199871994375ea1d8ba"}
Oct 30 12:46:16.413046 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:16.412675    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"
Oct 30 12:46:16.450542 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:16.450481    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm" podStartSLOduration=1.450465782 podStartE2EDuration="1.450465782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:16.449973646 +0000 UTC m=+564.974257865" watchObservedRunningTime="2026-10-30 12:46:16.450465782 +0000 UTC m=+564.974750001"
Oct 30 12:46:20.128879 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.128820    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"]
Oct 30 12:46:20.133022 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.132999    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"
Oct 30 12:46:20.162162 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.162132    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:46:20.162348 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.162224    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:46:20.163017 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.162994    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:46:20.163125 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.163109    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:46:20.164369 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.164345    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"]
Oct 30 12:46:20.175564 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.175520    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92ba3f8-821d-48d2-8da1-4fd819c736d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6\" (UID: \"e92ba3f8-821d-48d2-8da1-4fd819c736d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"
Oct 30 12:46:20.175700 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.175606    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvzh\" (UniqueName: \"kubernetes.io/projected/e92ba3f8-821d-48d2-8da1-4fd819c736d3-kube-api-access-7vvzh\") pod \"maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6\" (UID: \"e92ba3f8-821d-48d2-8da1-4fd819c736d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"
Oct 30 12:46:20.175862 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.175838    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92ba3f8-821d-48d2-8da1-4fd819c736d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6\" (UID: \"e92ba3f8-821d-48d2-8da1-4fd819c736d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"
Oct 30 12:46:20.276936 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.276890    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92ba3f8-821d-48d2-8da1-4fd819c736d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6\" (UID: \"e92ba3f8-821d-48d2-8da1-4fd819c736d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"
Oct 30 12:46:20.276936 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.276944    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvzh\" (UniqueName: \"kubernetes.io/projected/e92ba3f8-821d-48d2-8da1-4fd819c736d3-kube-api-access-7vvzh\") pod \"maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6\" (UID: \"e92ba3f8-821d-48d2-8da1-4fd819c736d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"
Oct 30 12:46:20.277202 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.277008    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92ba3f8-821d-48d2-8da1-4fd819c736d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6\" (UID: \"e92ba3f8-821d-48d2-8da1-4fd819c736d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"
Oct 30 12:46:20.277687 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.277666    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92ba3f8-821d-48d2-8da1-4fd819c736d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6\" (UID: \"e92ba3f8-821d-48d2-8da1-4fd819c736d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"
Oct 30 12:46:20.278542 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.278515    5837 patch_prober.go:29] interesting pod/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:46:20.278638 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.278558    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:46:20.279563 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.279530    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92ba3f8-821d-48d2-8da1-4fd819c736d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6\" (UID: \"e92ba3f8-821d-48d2-8da1-4fd819c736d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"
Oct 30 12:46:20.313763 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.313726    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvzh\" (UniqueName: \"kubernetes.io/projected/e92ba3f8-821d-48d2-8da1-4fd819c736d3-kube-api-access-7vvzh\") pod \"maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6\" (UID: \"e92ba3f8-821d-48d2-8da1-4fd819c736d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"
Oct 30 12:46:20.453404 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.453307    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"
Oct 30 12:46:20.771352 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:20.771321    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"]
Oct 30 12:46:20.773921 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:20.773881    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92ba3f8_821d_48d2_8da1_4fd819c736d3.slice/crio-f36049e9f41e401679ae3b16a6f4d051f02d92085f55f1cfa46937530c486e4e WatchSource:0}: Error finding container f36049e9f41e401679ae3b16a6f4d051f02d92085f55f1cfa46937530c486e4e: Status 404 returned error can't find the container with id f36049e9f41e401679ae3b16a6f4d051f02d92085f55f1cfa46937530c486e4e
Oct 30 12:46:21.453882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:21.453836    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6" event={"ID":"e92ba3f8-821d-48d2-8da1-4fd819c736d3","Type":"ContainerStarted","Data":"f36049e9f41e401679ae3b16a6f4d051f02d92085f55f1cfa46937530c486e4e"}
Oct 30 12:46:22.463033 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:22.463000    5837 generic.go:356] "Generic (PLEG): container finished" podID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerID="437c97d3095c2d56da056c948fb1374790617c10936928207b5807b1946bde5f" exitCode=0
Oct 30 12:46:22.463574 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:22.463067    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" event={"ID":"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4","Type":"ContainerDied","Data":"437c97d3095c2d56da056c948fb1374790617c10936928207b5807b1946bde5f"}
Oct 30 12:46:22.463574 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:22.463551    5837 scope.go:122] "RemoveContainer" containerID="437c97d3095c2d56da056c948fb1374790617c10936928207b5807b1946bde5f"
Oct 30 12:46:22.464928 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:22.464900    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:22.465021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:22.464968    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:22.465021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:22.465005    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:23.471153 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:23.471122    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" event={"ID":"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4","Type":"ContainerStarted","Data":"f207bd6d7df7a05e7a26d5d8e36f7ab5fef851e0d640adecb3dbfe6e1d1d7b7a"}
Oct 30 12:46:23.536995 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:23.536956    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:23.537191 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:23.537066    5837 patch_prober.go:29] interesting pod/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 12:46:23.537191 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:23.537099    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 12:46:24.380488 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:24.380064    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"
Oct 30 12:46:24.380685 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:24.380671    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"
Oct 30 12:46:24.381643 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:24.381619    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"
Oct 30 12:46:24.541902 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:24.540856    5837 patch_prober.go:29] interesting pod/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 12:46:24.541902 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:24.540932    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 12:46:25.390778 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.389617    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"
Oct 30 12:46:25.421361 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.421305    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"]
Oct 30 12:46:25.425687 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.425655    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"
Oct 30 12:46:25.438176 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.438011    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"]
Oct 30 12:46:25.472826 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.472574    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gg6p\" (UniqueName: \"kubernetes.io/projected/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-kube-api-access-7gg6p\") pod \"maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8\" (UID: \"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"
Oct 30 12:46:25.472826 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.472626    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8\" (UID: \"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"
Oct 30 12:46:25.472826 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.472653    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8\" (UID: \"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"
Oct 30 12:46:25.517470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.517431    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6" event={"ID":"e92ba3f8-821d-48d2-8da1-4fd819c736d3","Type":"ContainerStarted","Data":"5e6cd9925ce2c820d3d8eea86e0b5b182367b818f253fc8374f4512c81ba10b3"}
Oct 30 12:46:25.536519 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.535957    5837 patch_prober.go:29] interesting pod/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 12:46:25.536519 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.536010    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 12:46:25.558784 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.558739    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6" podStartSLOduration=1.875598546 podStartE2EDuration="5.558721915s" totalImagesPullingTime="3.683123369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:25.55678353 +0000 UTC m=+574.081067750" watchObservedRunningTime="2026-10-30 12:46:25.558721915 +0000 UTC m=+574.083006152"
Oct 30 12:46:25.573977 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.573947    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gg6p\" (UniqueName: \"kubernetes.io/projected/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-kube-api-access-7gg6p\") pod \"maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8\" (UID: \"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"
Oct 30 12:46:25.574134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.573989    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8\" (UID: \"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"
Oct 30 12:46:25.574134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.574005    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8\" (UID: \"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"
Oct 30 12:46:25.574774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.574750    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8\" (UID: \"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"
Oct 30 12:46:25.576738 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.576711    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8\" (UID: \"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"
Oct 30 12:46:25.613927 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.613896    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gg6p\" (UniqueName: \"kubernetes.io/projected/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-kube-api-access-7gg6p\") pod \"maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8\" (UID: \"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"
Oct 30 12:46:25.744813 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:25.744783    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"
Oct 30 12:46:26.063192 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:26.063157    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"]
Oct 30 12:46:26.064161 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:26.064134    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f30bd35_00c6_4ffa_a95c_ca9a11c4f58d.slice/crio-bd617ba33f0bcdf6c8e58ecf5e16ab1503b228db9e47de7f9a3c706da8bf136e WatchSource:0}: Error finding container bd617ba33f0bcdf6c8e58ecf5e16ab1503b228db9e47de7f9a3c706da8bf136e: Status 404 returned error can't find the container with id bd617ba33f0bcdf6c8e58ecf5e16ab1503b228db9e47de7f9a3c706da8bf136e
Oct 30 12:46:26.415871 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:26.415843    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"
Oct 30 12:46:26.526141 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:26.526113    5837 generic.go:356] "Generic (PLEG): container finished" podID="e92ba3f8-821d-48d2-8da1-4fd819c736d3" containerID="5e6cd9925ce2c820d3d8eea86e0b5b182367b818f253fc8374f4512c81ba10b3" exitCode=0
Oct 30 12:46:26.526327 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:26.526199    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6" event={"ID":"e92ba3f8-821d-48d2-8da1-4fd819c736d3","Type":"ContainerDied","Data":"5e6cd9925ce2c820d3d8eea86e0b5b182367b818f253fc8374f4512c81ba10b3"}
Oct 30 12:46:26.528026 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:26.527995    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8" event={"ID":"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d","Type":"ContainerStarted","Data":"84ba717f92a74e60cdfde81cffc45806fc3863221dbf8a1f5cc7068ce95786e4"}
Oct 30 12:46:26.528146 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:26.528039    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8" event={"ID":"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d","Type":"ContainerStarted","Data":"bd617ba33f0bcdf6c8e58ecf5e16ab1503b228db9e47de7f9a3c706da8bf136e"}
Oct 30 12:46:26.536876 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:26.536850    5837 patch_prober.go:29] interesting pod/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 12:46:26.536998 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:26.536914    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 12:46:26.610415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:26.610286    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8" podStartSLOduration=1.610254077 podStartE2EDuration="1.610254077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:26.609062685 +0000 UTC m=+575.133346904" watchObservedRunningTime="2026-10-30 12:46:26.610254077 +0000 UTC m=+575.134538297"
Oct 30 12:46:26.842327 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:26.842290    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"]
Oct 30 12:46:26.846973 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:26.846952    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"
Oct 30 12:46:26.864620 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:26.864551    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"]
Oct 30 12:46:26.913266 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:26.913234    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84fa4736-671e-4c79-b9ca-3909dc69449f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5431fa34-rml6w\" (UID: \"84fa4736-671e-4c79-b9ca-3909dc69449f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"
Oct 30 12:46:26.913514 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:26.913274    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84fa4736-671e-4c79-b9ca-3909dc69449f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5431fa34-rml6w\" (UID: \"84fa4736-671e-4c79-b9ca-3909dc69449f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"
Oct 30 12:46:26.913514 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:26.913479    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhx9\" (UniqueName: \"kubernetes.io/projected/84fa4736-671e-4c79-b9ca-3909dc69449f-kube-api-access-qjhx9\") pod \"maas-api-revoke-keys-e2e-ait-5431fa34-rml6w\" (UID: \"84fa4736-671e-4c79-b9ca-3909dc69449f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"
Oct 30 12:46:27.014900 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.014865    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhx9\" (UniqueName: \"kubernetes.io/projected/84fa4736-671e-4c79-b9ca-3909dc69449f-kube-api-access-qjhx9\") pod \"maas-api-revoke-keys-e2e-ait-5431fa34-rml6w\" (UID: \"84fa4736-671e-4c79-b9ca-3909dc69449f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"
Oct 30 12:46:27.015094 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.014991    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84fa4736-671e-4c79-b9ca-3909dc69449f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5431fa34-rml6w\" (UID: \"84fa4736-671e-4c79-b9ca-3909dc69449f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"
Oct 30 12:46:27.015094 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.015023    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84fa4736-671e-4c79-b9ca-3909dc69449f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5431fa34-rml6w\" (UID: \"84fa4736-671e-4c79-b9ca-3909dc69449f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"
Oct 30 12:46:27.015812 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.015745    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84fa4736-671e-4c79-b9ca-3909dc69449f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5431fa34-rml6w\" (UID: \"84fa4736-671e-4c79-b9ca-3909dc69449f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"
Oct 30 12:46:27.017521 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.017499    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84fa4736-671e-4c79-b9ca-3909dc69449f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5431fa34-rml6w\" (UID: \"84fa4736-671e-4c79-b9ca-3909dc69449f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"
Oct 30 12:46:27.033304 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.033277    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhx9\" (UniqueName: \"kubernetes.io/projected/84fa4736-671e-4c79-b9ca-3909dc69449f-kube-api-access-qjhx9\") pod \"maas-api-revoke-keys-e2e-ait-5431fa34-rml6w\" (UID: \"84fa4736-671e-4c79-b9ca-3909dc69449f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"
Oct 30 12:46:27.163220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.163184    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"
Oct 30 12:46:27.422017 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.421933    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"
Oct 30 12:46:27.517420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.515883    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"]
Oct 30 12:46:27.523505 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:27.523477    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fa4736_671e_4c79_b9ca_3909dc69449f.slice/crio-0e37fe0742b3a70d2f864beaef9eb6945d47eb8f967ee4b6464bfe3a5068df57 WatchSource:0}: Error finding container 0e37fe0742b3a70d2f864beaef9eb6945d47eb8f967ee4b6464bfe3a5068df57: Status 404 returned error can't find the container with id 0e37fe0742b3a70d2f864beaef9eb6945d47eb8f967ee4b6464bfe3a5068df57
Oct 30 12:46:27.535918 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.535891    5837 patch_prober.go:29] interesting pod/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 12:46:27.536038 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.535960    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 12:46:27.536671 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.536644    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w" event={"ID":"84fa4736-671e-4c79-b9ca-3909dc69449f","Type":"ContainerStarted","Data":"0e37fe0742b3a70d2f864beaef9eb6945d47eb8f967ee4b6464bfe3a5068df57"}
Oct 30 12:46:27.538713 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.538691    5837 generic.go:356] "Generic (PLEG): container finished" podID="3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d" containerID="84ba717f92a74e60cdfde81cffc45806fc3863221dbf8a1f5cc7068ce95786e4" exitCode=0
Oct 30 12:46:27.538806 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.538721    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8" event={"ID":"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d","Type":"ContainerDied","Data":"84ba717f92a74e60cdfde81cffc45806fc3863221dbf8a1f5cc7068ce95786e4"}
Oct 30 12:46:27.749083 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.749060    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"
Oct 30 12:46:27.830466 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.830426    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92ba3f8-821d-48d2-8da1-4fd819c736d3-kube-api-access-7vvzh\" (UniqueName: \"kubernetes.io/projected/e92ba3f8-821d-48d2-8da1-4fd819c736d3-kube-api-access-7vvzh\") pod \"e92ba3f8-821d-48d2-8da1-4fd819c736d3\" (UID: \"e92ba3f8-821d-48d2-8da1-4fd819c736d3\") "
Oct 30 12:46:27.830654 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.830619    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e92ba3f8-821d-48d2-8da1-4fd819c736d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92ba3f8-821d-48d2-8da1-4fd819c736d3-maas-api-xks-ca\") pod \"e92ba3f8-821d-48d2-8da1-4fd819c736d3\" (UID: \"e92ba3f8-821d-48d2-8da1-4fd819c736d3\") "
Oct 30 12:46:27.830744 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.830726    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92ba3f8-821d-48d2-8da1-4fd819c736d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92ba3f8-821d-48d2-8da1-4fd819c736d3-maas-api-openshift-ca\") pod \"e92ba3f8-821d-48d2-8da1-4fd819c736d3\" (UID: \"e92ba3f8-821d-48d2-8da1-4fd819c736d3\") "
Oct 30 12:46:27.831229 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.831145    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92ba3f8-821d-48d2-8da1-4fd819c736d3-maas-api-openshift-ca" pod "e92ba3f8-821d-48d2-8da1-4fd819c736d3" (UID: "e92ba3f8-821d-48d2-8da1-4fd819c736d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:27.832822 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.832794    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92ba3f8-821d-48d2-8da1-4fd819c736d3-kube-api-access-7vvzh" pod "e92ba3f8-821d-48d2-8da1-4fd819c736d3" (UID: "e92ba3f8-821d-48d2-8da1-4fd819c736d3"). InnerVolumeSpecName "kube-api-access-7vvzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:27.834566 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.834541    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92ba3f8-821d-48d2-8da1-4fd819c736d3-maas-api-xks-ca" pod "e92ba3f8-821d-48d2-8da1-4fd819c736d3" (UID: "e92ba3f8-821d-48d2-8da1-4fd819c736d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:27.932497 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.932469    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92ba3f8-821d-48d2-8da1-4fd819c736d3-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:27.932497 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.932496    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vvzh\" (UniqueName: \"kubernetes.io/projected/e92ba3f8-821d-48d2-8da1-4fd819c736d3-kube-api-access-7vvzh\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:27.932642 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:27.932505    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92ba3f8-821d-48d2-8da1-4fd819c736d3-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.537139 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:28.536521    5837 patch_prober.go:29] interesting pod/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 12:46:28.537139 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:28.536599    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 12:46:28.545675 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:28.545645    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"
Oct 30 12:46:28.545675 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:28.545660    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6" event={"ID":"e92ba3f8-821d-48d2-8da1-4fd819c736d3","Type":"ContainerDied","Data":"f36049e9f41e401679ae3b16a6f4d051f02d92085f55f1cfa46937530c486e4e"}
Oct 30 12:46:28.545882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:28.545719    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f36049e9f41e401679ae3b16a6f4d051f02d92085f55f1cfa46937530c486e4e"
Oct 30 12:46:28.547747 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:28.547715    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w" event={"ID":"84fa4736-671e-4c79-b9ca-3909dc69449f","Type":"ContainerStarted","Data":"546ddb2149cf28e253ac4a889eab7808d0ce5a2f3ddbb11925a76aad354c8cf9"}
Oct 30 12:46:28.607423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:28.605314    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w" podStartSLOduration=2.605293058 podStartE2EDuration="2.605293058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:28.591274739 +0000 UTC m=+577.115558958" watchObservedRunningTime="2026-10-30 12:46:28.605293058 +0000 UTC m=+577.129577277"
Oct 30 12:46:28.824696 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:28.824628    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"
Oct 30 12:46:28.945184 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:28.945150    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-maas-api-xks-ca\") pod \"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d\" (UID: \"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d\") "
Oct 30 12:46:28.945349 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:28.945211    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-kube-api-access-7gg6p\" (UniqueName: \"kubernetes.io/projected/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-kube-api-access-7gg6p\") pod \"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d\" (UID: \"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d\") "
Oct 30 12:46:28.945349 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:28.945317    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-maas-api-openshift-ca\") pod \"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d\" (UID: \"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d\") "
Oct 30 12:46:28.946098 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:28.946062    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-maas-api-openshift-ca" pod "3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d" (UID: "3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:28.947507 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:28.947480    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-maas-api-xks-ca" pod "3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d" (UID: "3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:28.948422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:28.948372    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-kube-api-access-7gg6p" pod "3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d" (UID: "3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d"). InnerVolumeSpecName "kube-api-access-7gg6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:29.046877 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:29.046843    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:29.046877 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:29.046872    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gg6p\" (UniqueName: \"kubernetes.io/projected/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-kube-api-access-7gg6p\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:29.046877 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:29.046883    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:29.540229 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:29.540197    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:29.540517 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:29.540493    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:29.541248 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:29.541226    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:29.555830 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:29.555799    5837 generic.go:356] "Generic (PLEG): container finished" podID="84fa4736-671e-4c79-b9ca-3909dc69449f" containerID="546ddb2149cf28e253ac4a889eab7808d0ce5a2f3ddbb11925a76aad354c8cf9" exitCode=0
Oct 30 12:46:29.555944 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:29.555873    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w" event={"ID":"84fa4736-671e-4c79-b9ca-3909dc69449f","Type":"ContainerDied","Data":"546ddb2149cf28e253ac4a889eab7808d0ce5a2f3ddbb11925a76aad354c8cf9"}
Oct 30 12:46:29.557462 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:29.557441    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8" event={"ID":"3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d","Type":"ContainerDied","Data":"bd617ba33f0bcdf6c8e58ecf5e16ab1503b228db9e47de7f9a3c706da8bf136e"}
Oct 30 12:46:29.557462 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:29.557462    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"
Oct 30 12:46:29.557581 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:29.557479    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd617ba33f0bcdf6c8e58ecf5e16ab1503b228db9e47de7f9a3c706da8bf136e"
Oct 30 12:46:30.564253 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.564221    5837 generic.go:356] "Generic (PLEG): container finished" podID="e42e36f7-20c2-4259-99c0-a221e144d330" containerID="df466c4ae60f81ed662c1b8738a26830e68a8702e3e6ba31b330ed4df0b255b9" exitCode=0
Oct 30 12:46:30.564734 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.564296    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7" event={"ID":"e42e36f7-20c2-4259-99c0-a221e144d330","Type":"ContainerDied","Data":"df466c4ae60f81ed662c1b8738a26830e68a8702e3e6ba31b330ed4df0b255b9"}
Oct 30 12:46:30.565070 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.564917    5837 scope.go:122] "RemoveContainer" containerID="df466c4ae60f81ed662c1b8738a26830e68a8702e3e6ba31b330ed4df0b255b9"
Oct 30 12:46:30.566236 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.566210    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:30.566314 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.566278    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:30.566314 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.566310    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:30.824575 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.824499    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"
Oct 30 12:46:30.867371 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.867342    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84fa4736-671e-4c79-b9ca-3909dc69449f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84fa4736-671e-4c79-b9ca-3909dc69449f-maas-api-xks-ca\") pod \"84fa4736-671e-4c79-b9ca-3909dc69449f\" (UID: \"84fa4736-671e-4c79-b9ca-3909dc69449f\") "
Oct 30 12:46:30.867589 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.867483    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fa4736-671e-4c79-b9ca-3909dc69449f-kube-api-access-qjhx9\" (UniqueName: \"kubernetes.io/projected/84fa4736-671e-4c79-b9ca-3909dc69449f-kube-api-access-qjhx9\") pod \"84fa4736-671e-4c79-b9ca-3909dc69449f\" (UID: \"84fa4736-671e-4c79-b9ca-3909dc69449f\") "
Oct 30 12:46:30.867589 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.867568    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84fa4736-671e-4c79-b9ca-3909dc69449f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84fa4736-671e-4c79-b9ca-3909dc69449f-maas-api-openshift-ca\") pod \"84fa4736-671e-4c79-b9ca-3909dc69449f\" (UID: \"84fa4736-671e-4c79-b9ca-3909dc69449f\") "
Oct 30 12:46:30.869124 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.869082    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fa4736-671e-4c79-b9ca-3909dc69449f-maas-api-openshift-ca" pod "84fa4736-671e-4c79-b9ca-3909dc69449f" (UID: "84fa4736-671e-4c79-b9ca-3909dc69449f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:30.879969 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.879927    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84fa4736-671e-4c79-b9ca-3909dc69449f-maas-api-xks-ca" pod "84fa4736-671e-4c79-b9ca-3909dc69449f" (UID: "84fa4736-671e-4c79-b9ca-3909dc69449f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:30.881261 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.881236    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fa4736-671e-4c79-b9ca-3909dc69449f-kube-api-access-qjhx9" pod "84fa4736-671e-4c79-b9ca-3909dc69449f" (UID: "84fa4736-671e-4c79-b9ca-3909dc69449f"). InnerVolumeSpecName "kube-api-access-qjhx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:30.938019 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.937985    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"]
Oct 30 12:46:30.938680 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.938662    5837 cpu_manager.go:414] "containerMap: removing container" podUID="3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d" containerName="revoke-keys"
Oct 30 12:46:30.938743 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.938683    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d" containerName="revoke-keys"
Oct 30 12:46:30.938743 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.938701    5837 cpu_manager.go:414] "containerMap: removing container" podUID="84fa4736-671e-4c79-b9ca-3909dc69449f" containerName="revoke-keys"
Oct 30 12:46:30.938743 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.938708    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fa4736-671e-4c79-b9ca-3909dc69449f" containerName="revoke-keys"
Oct 30 12:46:30.938743 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.938740    5837 cpu_manager.go:414] "containerMap: removing container" podUID="e92ba3f8-821d-48d2-8da1-4fd819c736d3" containerName="revoke-keys"
Oct 30 12:46:30.938883 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.938748    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92ba3f8-821d-48d2-8da1-4fd819c736d3" containerName="revoke-keys"
Oct 30 12:46:30.938883 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.938872    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d" containerName="revoke-keys"
Oct 30 12:46:30.938943 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.938889    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fa4736-671e-4c79-b9ca-3909dc69449f" containerName="revoke-keys"
Oct 30 12:46:30.938943 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.938903    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92ba3f8-821d-48d2-8da1-4fd819c736d3" containerName="revoke-keys"
Oct 30 12:46:30.942951 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.942889    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"
Oct 30 12:46:30.959032 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.959009    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"]
Oct 30 12:46:30.969806 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.969777    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqv7r\" (UniqueName: \"kubernetes.io/projected/6b9df847-1b4f-42b8-ac07-864e775abf92-kube-api-access-lqv7r\") pod \"maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns\" (UID: \"6b9df847-1b4f-42b8-ac07-864e775abf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"
Oct 30 12:46:30.969945 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.969814    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9df847-1b4f-42b8-ac07-864e775abf92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns\" (UID: \"6b9df847-1b4f-42b8-ac07-864e775abf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"
Oct 30 12:46:30.969945 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.969849    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9df847-1b4f-42b8-ac07-864e775abf92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns\" (UID: \"6b9df847-1b4f-42b8-ac07-864e775abf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"
Oct 30 12:46:30.969945 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.969904    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84fa4736-671e-4c79-b9ca-3909dc69449f-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:30.969945 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.969914    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84fa4736-671e-4c79-b9ca-3909dc69449f-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:30.969945 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:30.969923    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjhx9\" (UniqueName: \"kubernetes.io/projected/84fa4736-671e-4c79-b9ca-3909dc69449f-kube-api-access-qjhx9\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:31.071677 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:31.071639    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqv7r\" (UniqueName: \"kubernetes.io/projected/6b9df847-1b4f-42b8-ac07-864e775abf92-kube-api-access-lqv7r\") pod \"maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns\" (UID: \"6b9df847-1b4f-42b8-ac07-864e775abf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"
Oct 30 12:46:31.071860 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:31.071695    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9df847-1b4f-42b8-ac07-864e775abf92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns\" (UID: \"6b9df847-1b4f-42b8-ac07-864e775abf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"
Oct 30 12:46:31.071860 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:31.071749    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9df847-1b4f-42b8-ac07-864e775abf92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns\" (UID: \"6b9df847-1b4f-42b8-ac07-864e775abf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"
Oct 30 12:46:31.072615 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:31.072588    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9df847-1b4f-42b8-ac07-864e775abf92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns\" (UID: \"6b9df847-1b4f-42b8-ac07-864e775abf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"
Oct 30 12:46:31.075492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:31.075436    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9df847-1b4f-42b8-ac07-864e775abf92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns\" (UID: \"6b9df847-1b4f-42b8-ac07-864e775abf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"
Oct 30 12:46:31.101371 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:31.099003    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqv7r\" (UniqueName: \"kubernetes.io/projected/6b9df847-1b4f-42b8-ac07-864e775abf92-kube-api-access-lqv7r\") pod \"maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns\" (UID: \"6b9df847-1b4f-42b8-ac07-864e775abf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"
Oct 30 12:46:31.264932 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:31.264903    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"
Oct 30 12:46:31.573413 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:31.573358    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7" event={"ID":"e42e36f7-20c2-4259-99c0-a221e144d330","Type":"ContainerStarted","Data":"c179fad227e65b4b4831f1e6675675095fd26b9bdcbb703925006215a02b6e6c"}
Oct 30 12:46:31.574717 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:31.574611    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"
Oct 30 12:46:31.574717 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:31.574644    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w" event={"ID":"84fa4736-671e-4c79-b9ca-3909dc69449f","Type":"ContainerDied","Data":"0e37fe0742b3a70d2f864beaef9eb6945d47eb8f967ee4b6464bfe3a5068df57"}
Oct 30 12:46:31.574717 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:31.574688    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e37fe0742b3a70d2f864beaef9eb6945d47eb8f967ee4b6464bfe3a5068df57"
Oct 30 12:46:31.652258 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:31.652165    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"]
Oct 30 12:46:31.709440 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:31.707658    5837 patch_prober.go:29] interesting pod/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 12:46:31.709440 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:31.707724    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7" podUID="e42e36f7-20c2-4259-99c0-a221e144d330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 12:46:31.728430 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:31.725361    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:32.582488 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:32.582450    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns" event={"ID":"6b9df847-1b4f-42b8-ac07-864e775abf92","Type":"ContainerStarted","Data":"da2512c7944a1f2f71a99fc107ebc59a819401f76510f3162b1e922171f1d44f"}
Oct 30 12:46:32.583021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:32.582519    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns" event={"ID":"6b9df847-1b4f-42b8-ac07-864e775abf92","Type":"ContainerStarted","Data":"32473876df47abc167a2f0206bc500349d38b3708a2b62c77bc009e9737f77b6"}
Oct 30 12:46:32.682333 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:32.682289    5837 patch_prober.go:29] interesting pod/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 12:46:32.682562 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:32.682344    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7" podUID="e42e36f7-20c2-4259-99c0-a221e144d330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 12:46:33.636005 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:33.635972    5837 generic.go:356] "Generic (PLEG): container finished" podID="6b9df847-1b4f-42b8-ac07-864e775abf92" containerID="da2512c7944a1f2f71a99fc107ebc59a819401f76510f3162b1e922171f1d44f" exitCode=0
Oct 30 12:46:33.637679 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:33.637642    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns" event={"ID":"6b9df847-1b4f-42b8-ac07-864e775abf92","Type":"ContainerDied","Data":"da2512c7944a1f2f71a99fc107ebc59a819401f76510f3162b1e922171f1d44f"}
Oct 30 12:46:33.705334 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:33.705299    5837 patch_prober.go:29] interesting pod/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 12:46:33.705639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:33.705596    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7" podUID="e42e36f7-20c2-4259-99c0-a221e144d330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 12:46:34.682955 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:34.682920    5837 patch_prober.go:29] interesting pod/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 12:46:34.683489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:34.682981    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7" podUID="e42e36f7-20c2-4259-99c0-a221e144d330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 12:46:34.842922 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:34.842892    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"
Oct 30 12:46:34.867230 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:34.867197    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b9df847-1b4f-42b8-ac07-864e775abf92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9df847-1b4f-42b8-ac07-864e775abf92-maas-api-openshift-ca\") pod \"6b9df847-1b4f-42b8-ac07-864e775abf92\" (UID: \"6b9df847-1b4f-42b8-ac07-864e775abf92\") "
Oct 30 12:46:34.867420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:34.867284    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9df847-1b4f-42b8-ac07-864e775abf92-kube-api-access-lqv7r\" (UniqueName: \"kubernetes.io/projected/6b9df847-1b4f-42b8-ac07-864e775abf92-kube-api-access-lqv7r\") pod \"6b9df847-1b4f-42b8-ac07-864e775abf92\" (UID: \"6b9df847-1b4f-42b8-ac07-864e775abf92\") "
Oct 30 12:46:34.867420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:34.867348    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b9df847-1b4f-42b8-ac07-864e775abf92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9df847-1b4f-42b8-ac07-864e775abf92-maas-api-xks-ca\") pod \"6b9df847-1b4f-42b8-ac07-864e775abf92\" (UID: \"6b9df847-1b4f-42b8-ac07-864e775abf92\") "
Oct 30 12:46:34.867670 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:34.867645    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b9df847-1b4f-42b8-ac07-864e775abf92-maas-api-openshift-ca" pod "6b9df847-1b4f-42b8-ac07-864e775abf92" (UID: "6b9df847-1b4f-42b8-ac07-864e775abf92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:34.867886 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:34.867868    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9df847-1b4f-42b8-ac07-864e775abf92-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:34.869718 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:34.869687    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9df847-1b4f-42b8-ac07-864e775abf92-kube-api-access-lqv7r" pod "6b9df847-1b4f-42b8-ac07-864e775abf92" (UID: "6b9df847-1b4f-42b8-ac07-864e775abf92"). InnerVolumeSpecName "kube-api-access-lqv7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:34.869825 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:34.869803    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b9df847-1b4f-42b8-ac07-864e775abf92-maas-api-xks-ca" pod "6b9df847-1b4f-42b8-ac07-864e775abf92" (UID: "6b9df847-1b4f-42b8-ac07-864e775abf92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:34.969220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:34.969135    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqv7r\" (UniqueName: \"kubernetes.io/projected/6b9df847-1b4f-42b8-ac07-864e775abf92-kube-api-access-lqv7r\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:34.969220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:34.969178    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9df847-1b4f-42b8-ac07-864e775abf92-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:35.648865 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:35.648829    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns" event={"ID":"6b9df847-1b4f-42b8-ac07-864e775abf92","Type":"ContainerDied","Data":"32473876df47abc167a2f0206bc500349d38b3708a2b62c77bc009e9737f77b6"}
Oct 30 12:46:35.649029 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:35.648892    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32473876df47abc167a2f0206bc500349d38b3708a2b62c77bc009e9737f77b6"
Oct 30 12:46:35.649029 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:35.648844    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"
Oct 30 12:46:35.683072 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:35.683039    5837 patch_prober.go:29] interesting pod/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 12:46:35.683545 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:35.683091    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7" podUID="e42e36f7-20c2-4259-99c0-a221e144d330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 12:46:35.902505 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:35.895222    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"]
Oct 30 12:46:36.249217 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:36.249187    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:36.682547 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:36.682462    5837 patch_prober.go:29] interesting pod/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 12:46:36.682547 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:36.682517    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7" podUID="e42e36f7-20c2-4259-99c0-a221e144d330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 12:46:37.290758 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:37.290728    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"]
Oct 30 12:46:37.318489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:37.318457    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"]
Oct 30 12:46:37.688117 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:37.688047    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:37.689181 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:37.689163    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:38.006187 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.006155    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"]
Oct 30 12:46:38.008177 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.006707    5837 cpu_manager.go:414] "containerMap: removing container" podUID="6b9df847-1b4f-42b8-ac07-864e775abf92" containerName="revoke-keys"
Oct 30 12:46:38.008177 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.006719    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9df847-1b4f-42b8-ac07-864e775abf92" containerName="revoke-keys"
Oct 30 12:46:38.008177 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.006790    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9df847-1b4f-42b8-ac07-864e775abf92" containerName="revoke-keys"
Oct 30 12:46:38.009869 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.009855    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"
Oct 30 12:46:38.024816 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.024795    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:46:38.024926 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.024812    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:46:38.024926 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.024853    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:46:38.024926 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.024920    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:46:38.027374 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.027353    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"]
Oct 30 12:46:38.122237 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.122203    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trlqr\" (UniqueName: \"kubernetes.io/projected/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-kube-api-access-trlqr\") pod \"maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n\" (UID: \"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"
Oct 30 12:46:38.122412 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.122251    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n\" (UID: \"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"
Oct 30 12:46:38.122412 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.122345    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n\" (UID: \"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"
Oct 30 12:46:38.223329 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.223253    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trlqr\" (UniqueName: \"kubernetes.io/projected/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-kube-api-access-trlqr\") pod \"maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n\" (UID: \"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"
Oct 30 12:46:38.223329 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.223292    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n\" (UID: \"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"
Oct 30 12:46:38.223528 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.223333    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n\" (UID: \"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"
Oct 30 12:46:38.223993 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.223964    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n\" (UID: \"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"
Oct 30 12:46:38.225814 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.225795    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n\" (UID: \"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"
Oct 30 12:46:38.241902 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.241878    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trlqr\" (UniqueName: \"kubernetes.io/projected/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-kube-api-access-trlqr\") pod \"maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n\" (UID: \"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"
Oct 30 12:46:38.327277 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.327200    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"
Oct 30 12:46:38.628546 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.628520    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"]
Oct 30 12:46:38.630627 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:38.630596    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b6d32a_6c5b_4610_8d4b_8659cf2bb49b.slice/crio-fc10f0715ab9781746ea51f72ef23a53587d3cdf66d7dc974d2635a9559bab96 WatchSource:0}: Error finding container fc10f0715ab9781746ea51f72ef23a53587d3cdf66d7dc974d2635a9559bab96: Status 404 returned error can't find the container with id fc10f0715ab9781746ea51f72ef23a53587d3cdf66d7dc974d2635a9559bab96
Oct 30 12:46:38.666637 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.666608    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n" event={"ID":"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b","Type":"ContainerStarted","Data":"fc10f0715ab9781746ea51f72ef23a53587d3cdf66d7dc974d2635a9559bab96"}
Oct 30 12:46:38.668518 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.668494    5837 generic.go:356] "Generic (PLEG): container finished" podID="147b05ae-0fc7-49bd-8521-8a092279d46c" containerID="ef05778be2425a7cab00379c1d0898b7eb34f711a9e65a183cdc89a4479d8315" exitCode=0
Oct 30 12:46:38.668641 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.668565    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd" event={"ID":"147b05ae-0fc7-49bd-8521-8a092279d46c","Type":"ContainerDied","Data":"ef05778be2425a7cab00379c1d0898b7eb34f711a9e65a183cdc89a4479d8315"}
Oct 30 12:46:38.669126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.669106    5837 scope.go:122] "RemoveContainer" containerID="ef05778be2425a7cab00379c1d0898b7eb34f711a9e65a183cdc89a4479d8315"
Oct 30 12:46:38.670310 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.670279    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:38.670386 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.670353    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:38.670456 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:38.670386    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:39.103169 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:39.103131    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"]
Oct 30 12:46:39.105792 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:39.105765    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdd5b3ae-lw6v6"]
Oct 30 12:46:39.315343 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:39.315311    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"]
Oct 30 12:46:39.315581 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:39.315539    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerName="istio-proxy" containerID="cri-o://f207bd6d7df7a05e7a26d5d8e36f7ab5fef851e0d640adecb3dbfe6e1d1d7b7a" gracePeriod=30
Oct 30 12:46:39.616896 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:39.616817    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e92ba3f8-821d-48d2-8da1-4fd819c736d3" path="/var/lib/kubelet/pods/e92ba3f8-821d-48d2-8da1-4fd819c736d3/volumes"
Oct 30 12:46:39.630922 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:39.630895    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"]
Oct 30 12:46:39.633163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:39.633142    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4da58cb4-g6vg8"]
Oct 30 12:46:39.683350 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:39.683310    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd" event={"ID":"147b05ae-0fc7-49bd-8521-8a092279d46c","Type":"ContainerStarted","Data":"79f1e5902772b25f8f96fa0c03cdf24ee7974e43f40ba1600dc8b404f3227231"}
Oct 30 12:46:39.705416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:39.696999    5837 generic.go:356] "Generic (PLEG): container finished" podID="f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b" containerID="83b67b47e8a5ca7c02167f1c636faf2d905f3b6f247258aa507dee2268077310" exitCode=0
Oct 30 12:46:39.705416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:39.697109    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n" event={"ID":"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b","Type":"ContainerDied","Data":"83b67b47e8a5ca7c02167f1c636faf2d905f3b6f247258aa507dee2268077310"}
Oct 30 12:46:40.066410 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:40.066361    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"]
Oct 30 12:46:40.070186 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:40.070164    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5431fa34-rml6w"]
Oct 30 12:46:40.529112 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:40.528255    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:40.530537 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:40.530502    5837 patch_prober.go:29] interesting pod/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:46:40.530676 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:40.530606    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd" podUID="147b05ae-0fc7-49bd-8521-8a092279d46c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 12:46:40.922974 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:40.922913    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q" podUID="834dc833-1f8e-439c-a15d-09bf0f851fe7" containerName="payload-pre-processing" containerID="cri-o://0d05fb92af6cd3b46d6c7659fcfc115009ec79938f4384be353eef4a1c03f06c" gracePeriod=25
Oct 30 12:46:40.931179 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:40.931144    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"
Oct 30 12:46:40.954993 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:40.954970    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-maas-api-xks-ca\") pod \"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b\" (UID: \"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b\") "
Oct 30 12:46:40.955102 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:40.955005    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-maas-api-openshift-ca\") pod \"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b\" (UID: \"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b\") "
Oct 30 12:46:40.955139 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:40.955100    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-kube-api-access-trlqr\" (UniqueName: \"kubernetes.io/projected/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-kube-api-access-trlqr\") pod \"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b\" (UID: \"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b\") "
Oct 30 12:46:40.955451 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:40.955424    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-maas-api-openshift-ca" pod "f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b" (UID: "f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:40.955585 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:40.955569    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:40.957171 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:40.957147    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-maas-api-xks-ca" pod "f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b" (UID: "f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:40.957242 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:40.957190    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-kube-api-access-trlqr" pod "f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b" (UID: "f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b"). InnerVolumeSpecName "kube-api-access-trlqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:41.057161 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.057126    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:41.057161 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.057157    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trlqr\" (UniqueName: \"kubernetes.io/projected/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b-kube-api-access-trlqr\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:41.246738 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.246707    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"
Oct 30 12:46:41.360619 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.360533    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/834dc833-1f8e-439c-a15d-09bf0f851fe7-kube-api-access-zdr8s\" (UniqueName: \"kubernetes.io/projected/834dc833-1f8e-439c-a15d-09bf0f851fe7-kube-api-access-zdr8s\") pod \"834dc833-1f8e-439c-a15d-09bf0f851fe7\" (UID: \"834dc833-1f8e-439c-a15d-09bf0f851fe7\") "
Oct 30 12:46:41.360795 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.360775    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/834dc833-1f8e-439c-a15d-09bf0f851fe7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834dc833-1f8e-439c-a15d-09bf0f851fe7-plugins-config-volume\") pod \"834dc833-1f8e-439c-a15d-09bf0f851fe7\" (UID: \"834dc833-1f8e-439c-a15d-09bf0f851fe7\") "
Oct 30 12:46:41.361144 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.361121    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/834dc833-1f8e-439c-a15d-09bf0f851fe7-plugins-config-volume" pod "834dc833-1f8e-439c-a15d-09bf0f851fe7" (UID: "834dc833-1f8e-439c-a15d-09bf0f851fe7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:41.362792 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.362773    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834dc833-1f8e-439c-a15d-09bf0f851fe7-kube-api-access-zdr8s" pod "834dc833-1f8e-439c-a15d-09bf0f851fe7" (UID: "834dc833-1f8e-439c-a15d-09bf0f851fe7"). InnerVolumeSpecName "kube-api-access-zdr8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:41.410942 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.410907    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"]
Oct 30 12:46:41.429789 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.429763    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"]
Oct 30 12:46:41.462738 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.462706    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834dc833-1f8e-439c-a15d-09bf0f851fe7-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:41.462738 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.462738    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdr8s\" (UniqueName: \"kubernetes.io/projected/834dc833-1f8e-439c-a15d-09bf0f851fe7-kube-api-access-zdr8s\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:41.528448 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.528410    5837 patch_prober.go:29] interesting pod/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:46:41.528639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.528473    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd" podUID="147b05ae-0fc7-49bd-8521-8a092279d46c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 12:46:41.614082 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.613996    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d" path="/var/lib/kubelet/pods/3f30bd35-00c6-4ffa-a95c-ca9a11c4f58d/volumes"
Oct 30 12:46:41.614533 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.614511    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84fa4736-671e-4c79-b9ca-3909dc69449f" path="/var/lib/kubelet/pods/84fa4736-671e-4c79-b9ca-3909dc69449f/volumes"
Oct 30 12:46:41.722173 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.722139    5837 generic.go:356] "Generic (PLEG): container finished" podID="834dc833-1f8e-439c-a15d-09bf0f851fe7" containerID="0d05fb92af6cd3b46d6c7659fcfc115009ec79938f4384be353eef4a1c03f06c" exitCode=0
Oct 30 12:46:41.722366 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.722188    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"
Oct 30 12:46:41.722366 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.722216    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q" event={"ID":"834dc833-1f8e-439c-a15d-09bf0f851fe7","Type":"ContainerDied","Data":"0d05fb92af6cd3b46d6c7659fcfc115009ec79938f4384be353eef4a1c03f06c"}
Oct 30 12:46:41.722366 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.722292    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q" event={"ID":"834dc833-1f8e-439c-a15d-09bf0f851fe7","Type":"ContainerDied","Data":"8fbc6fd46364c3d496b484cce115445af4e6a484e6123822247e504dcec33753"}
Oct 30 12:46:41.722366 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.722321    5837 scope.go:122] "RemoveContainer" containerID="0d05fb92af6cd3b46d6c7659fcfc115009ec79938f4384be353eef4a1c03f06c"
Oct 30 12:46:41.724338 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.724148    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n" event={"ID":"f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b","Type":"ContainerDied","Data":"fc10f0715ab9781746ea51f72ef23a53587d3cdf66d7dc974d2635a9559bab96"}
Oct 30 12:46:41.724338 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.724192    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc10f0715ab9781746ea51f72ef23a53587d3cdf66d7dc974d2635a9559bab96"
Oct 30 12:46:41.724338 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.724251    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"
Oct 30 12:46:41.733961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.733938    5837 scope.go:122] "RemoveContainer" containerID="0d05fb92af6cd3b46d6c7659fcfc115009ec79938f4384be353eef4a1c03f06c"
Oct 30 12:46:41.734253 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:46:41.734235    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d05fb92af6cd3b46d6c7659fcfc115009ec79938f4384be353eef4a1c03f06c\": container with ID starting with 0d05fb92af6cd3b46d6c7659fcfc115009ec79938f4384be353eef4a1c03f06c not found: ID does not exist" containerID="0d05fb92af6cd3b46d6c7659fcfc115009ec79938f4384be353eef4a1c03f06c"
Oct 30 12:46:41.734311 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.734262    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d05fb92af6cd3b46d6c7659fcfc115009ec79938f4384be353eef4a1c03f06c"} err="failed to get container status \"0d05fb92af6cd3b46d6c7659fcfc115009ec79938f4384be353eef4a1c03f06c\": rpc error: code = NotFound desc = could not find container \"0d05fb92af6cd3b46d6c7659fcfc115009ec79938f4384be353eef4a1c03f06c\": container with ID starting with 0d05fb92af6cd3b46d6c7659fcfc115009ec79938f4384be353eef4a1c03f06c not found: ID does not exist"
Oct 30 12:46:41.777229 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.777199    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"]
Oct 30 12:46:41.781749 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:41.781722    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4da58cb4-6568cfb4ff-ljv2q"]
Oct 30 12:46:42.294607 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.294528    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn" podUID="5bfec5e9-47b4-4090-a6ef-69172ebbf69f" containerName="payload-processing" containerID="cri-o://e23d76598b85b22b629117147edd20be3e2a803290e5c865ae14a446497aa63b" gracePeriod=25
Oct 30 12:46:42.319685 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.319625    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf" podUID="111eeb2b-43df-4895-a5d5-b101432ac848" containerName="payload-pre-processing" containerID="cri-o://c5775f9213884e37b0b0f77e0ac7b27e03a85b83778a9092e1a1fe29bad9a578" gracePeriod=25
Oct 30 12:46:42.532278 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.532218    5837 patch_prober.go:29] interesting pod/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:46:42.534661 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.532616    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd" podUID="147b05ae-0fc7-49bd-8521-8a092279d46c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 12:46:42.618718 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.618690    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"
Oct 30 12:46:42.658240 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.658214    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"
Oct 30 12:46:42.678168 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.678141    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfec5e9-47b4-4090-a6ef-69172ebbf69f-kube-api-access-8bbpw\" (UniqueName: \"kubernetes.io/projected/5bfec5e9-47b4-4090-a6ef-69172ebbf69f-kube-api-access-8bbpw\") pod \"5bfec5e9-47b4-4090-a6ef-69172ebbf69f\" (UID: \"5bfec5e9-47b4-4090-a6ef-69172ebbf69f\") "
Oct 30 12:46:42.678326 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.678180    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bfec5e9-47b4-4090-a6ef-69172ebbf69f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bfec5e9-47b4-4090-a6ef-69172ebbf69f-plugins-config-volume\") pod \"5bfec5e9-47b4-4090-a6ef-69172ebbf69f\" (UID: \"5bfec5e9-47b4-4090-a6ef-69172ebbf69f\") "
Oct 30 12:46:42.678641 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.678611    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfec5e9-47b4-4090-a6ef-69172ebbf69f-plugins-config-volume" pod "5bfec5e9-47b4-4090-a6ef-69172ebbf69f" (UID: "5bfec5e9-47b4-4090-a6ef-69172ebbf69f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:42.680549 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.680525    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfec5e9-47b4-4090-a6ef-69172ebbf69f-kube-api-access-8bbpw" pod "5bfec5e9-47b4-4090-a6ef-69172ebbf69f" (UID: "5bfec5e9-47b4-4090-a6ef-69172ebbf69f"). InnerVolumeSpecName "kube-api-access-8bbpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:42.732069 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.732038    5837 generic.go:356] "Generic (PLEG): container finished" podID="5bfec5e9-47b4-4090-a6ef-69172ebbf69f" containerID="e23d76598b85b22b629117147edd20be3e2a803290e5c865ae14a446497aa63b" exitCode=0
Oct 30 12:46:42.732265 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.732093    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"
Oct 30 12:46:42.732265 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.732124    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn" event={"ID":"5bfec5e9-47b4-4090-a6ef-69172ebbf69f","Type":"ContainerDied","Data":"e23d76598b85b22b629117147edd20be3e2a803290e5c865ae14a446497aa63b"}
Oct 30 12:46:42.732265 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.732188    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn" event={"ID":"5bfec5e9-47b4-4090-a6ef-69172ebbf69f","Type":"ContainerDied","Data":"df47d3ae42f2a1c10bd8f699025a341f2d695a7a4bd5b1c65fa73d2505040681"}
Oct 30 12:46:42.732265 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.732218    5837 scope.go:122] "RemoveContainer" containerID="e23d76598b85b22b629117147edd20be3e2a803290e5c865ae14a446497aa63b"
Oct 30 12:46:42.734002 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.733841    5837 generic.go:356] "Generic (PLEG): container finished" podID="111eeb2b-43df-4895-a5d5-b101432ac848" containerID="c5775f9213884e37b0b0f77e0ac7b27e03a85b83778a9092e1a1fe29bad9a578" exitCode=0
Oct 30 12:46:42.734002 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.733914    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf" event={"ID":"111eeb2b-43df-4895-a5d5-b101432ac848","Type":"ContainerDied","Data":"c5775f9213884e37b0b0f77e0ac7b27e03a85b83778a9092e1a1fe29bad9a578"}
Oct 30 12:46:42.734002 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.733961    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf" event={"ID":"111eeb2b-43df-4895-a5d5-b101432ac848","Type":"ContainerDied","Data":"43ce0520b81686efb8240bfeb3607550f425f1c3534fff07a6e51289c5312b97"}
Oct 30 12:46:42.734002 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.733920    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"
Oct 30 12:46:42.742538 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.742511    5837 scope.go:122] "RemoveContainer" containerID="e23d76598b85b22b629117147edd20be3e2a803290e5c865ae14a446497aa63b"
Oct 30 12:46:42.742873 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:46:42.742849    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e23d76598b85b22b629117147edd20be3e2a803290e5c865ae14a446497aa63b\": container with ID starting with e23d76598b85b22b629117147edd20be3e2a803290e5c865ae14a446497aa63b not found: ID does not exist" containerID="e23d76598b85b22b629117147edd20be3e2a803290e5c865ae14a446497aa63b"
Oct 30 12:46:42.742966 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.742879    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e23d76598b85b22b629117147edd20be3e2a803290e5c865ae14a446497aa63b"} err="failed to get container status \"e23d76598b85b22b629117147edd20be3e2a803290e5c865ae14a446497aa63b\": rpc error: code = NotFound desc = could not find container \"e23d76598b85b22b629117147edd20be3e2a803290e5c865ae14a446497aa63b\": container with ID starting with e23d76598b85b22b629117147edd20be3e2a803290e5c865ae14a446497aa63b not found: ID does not exist"
Oct 30 12:46:42.742966 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.742896    5837 scope.go:122] "RemoveContainer" containerID="c5775f9213884e37b0b0f77e0ac7b27e03a85b83778a9092e1a1fe29bad9a578"
Oct 30 12:46:42.751965 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.751940    5837 scope.go:122] "RemoveContainer" containerID="c5775f9213884e37b0b0f77e0ac7b27e03a85b83778a9092e1a1fe29bad9a578"
Oct 30 12:46:42.752222 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:46:42.752205    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5775f9213884e37b0b0f77e0ac7b27e03a85b83778a9092e1a1fe29bad9a578\": container with ID starting with c5775f9213884e37b0b0f77e0ac7b27e03a85b83778a9092e1a1fe29bad9a578 not found: ID does not exist" containerID="c5775f9213884e37b0b0f77e0ac7b27e03a85b83778a9092e1a1fe29bad9a578"
Oct 30 12:46:42.752267 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.752228    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5775f9213884e37b0b0f77e0ac7b27e03a85b83778a9092e1a1fe29bad9a578"} err="failed to get container status \"c5775f9213884e37b0b0f77e0ac7b27e03a85b83778a9092e1a1fe29bad9a578\": rpc error: code = NotFound desc = could not find container \"c5775f9213884e37b0b0f77e0ac7b27e03a85b83778a9092e1a1fe29bad9a578\": container with ID starting with c5775f9213884e37b0b0f77e0ac7b27e03a85b83778a9092e1a1fe29bad9a578 not found: ID does not exist"
Oct 30 12:46:42.779067 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.779045    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/111eeb2b-43df-4895-a5d5-b101432ac848-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111eeb2b-43df-4895-a5d5-b101432ac848-plugins-config-volume\") pod \"111eeb2b-43df-4895-a5d5-b101432ac848\" (UID: \"111eeb2b-43df-4895-a5d5-b101432ac848\") "
Oct 30 12:46:42.779213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.779195    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111eeb2b-43df-4895-a5d5-b101432ac848-kube-api-access-ctnb7\" (UniqueName: \"kubernetes.io/projected/111eeb2b-43df-4895-a5d5-b101432ac848-kube-api-access-ctnb7\") pod \"111eeb2b-43df-4895-a5d5-b101432ac848\" (UID: \"111eeb2b-43df-4895-a5d5-b101432ac848\") "
Oct 30 12:46:42.779444 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.779422    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111eeb2b-43df-4895-a5d5-b101432ac848-plugins-config-volume" pod "111eeb2b-43df-4895-a5d5-b101432ac848" (UID: "111eeb2b-43df-4895-a5d5-b101432ac848"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:42.779693 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.779677    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bbpw\" (UniqueName: \"kubernetes.io/projected/5bfec5e9-47b4-4090-a6ef-69172ebbf69f-kube-api-access-8bbpw\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:42.779752 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.779699    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bfec5e9-47b4-4090-a6ef-69172ebbf69f-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:42.779752 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.779714    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111eeb2b-43df-4895-a5d5-b101432ac848-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:42.781308 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.781288    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111eeb2b-43df-4895-a5d5-b101432ac848-kube-api-access-ctnb7" pod "111eeb2b-43df-4895-a5d5-b101432ac848" (UID: "111eeb2b-43df-4895-a5d5-b101432ac848"). InnerVolumeSpecName "kube-api-access-ctnb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:42.813162 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.813094    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"]
Oct 30 12:46:42.815931 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.815900    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5431fa34-78fdf5698d-tg8rn"]
Oct 30 12:46:42.880671 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.880644    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctnb7\" (UniqueName: \"kubernetes.io/projected/111eeb2b-43df-4895-a5d5-b101432ac848-kube-api-access-ctnb7\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:42.946150 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.946115    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"]
Oct 30 12:46:42.949353 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:42.949326    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1b2ed0a-mxxns"]
Oct 30 12:46:43.077001 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:43.076970    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"]
Oct 30 12:46:43.080349 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:43.080315    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5431fa34-75999fc7c9-s5dbf"]
Oct 30 12:46:43.368468 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:43.368440    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:43.528718 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:43.528692    5837 patch_prober.go:29] interesting pod/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:46:43.528888 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:43.528748    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd" podUID="147b05ae-0fc7-49bd-8521-8a092279d46c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 12:46:43.614375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:43.614345    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="111eeb2b-43df-4895-a5d5-b101432ac848" path="/var/lib/kubelet/pods/111eeb2b-43df-4895-a5d5-b101432ac848/volumes"
Oct 30 12:46:43.614747 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:43.614732    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bfec5e9-47b4-4090-a6ef-69172ebbf69f" path="/var/lib/kubelet/pods/5bfec5e9-47b4-4090-a6ef-69172ebbf69f/volumes"
Oct 30 12:46:43.615076 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:43.615062    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b9df847-1b4f-42b8-ac07-864e775abf92" path="/var/lib/kubelet/pods/6b9df847-1b4f-42b8-ac07-864e775abf92/volumes"
Oct 30 12:46:43.615699 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:43.615684    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="834dc833-1f8e-439c-a15d-09bf0f851fe7" path="/var/lib/kubelet/pods/834dc833-1f8e-439c-a15d-09bf0f851fe7/volumes"
Oct 30 12:46:44.529419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.528506    5837 patch_prober.go:29] interesting pod/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:46:44.529419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.528564    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd" podUID="147b05ae-0fc7-49bd-8521-8a092279d46c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 12:46:44.542525 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.542505    5837 patch_prober.go:29] interesting pod/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 12:46:44.542596 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.542548    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 12:46:44.663056 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.663030    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:44.750388 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.750357    5837 generic.go:356] "Generic (PLEG): container finished" podID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerID="f207bd6d7df7a05e7a26d5d8e36f7ab5fef851e0d640adecb3dbfe6e1d1d7b7a" exitCode=0
Oct 30 12:46:44.750591 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.750432    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"
Oct 30 12:46:44.750591 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.750435    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" event={"ID":"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4","Type":"ContainerDied","Data":"f207bd6d7df7a05e7a26d5d8e36f7ab5fef851e0d640adecb3dbfe6e1d1d7b7a"}
Oct 30 12:46:44.750591 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.750494    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g" event={"ID":"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4","Type":"ContainerDied","Data":"cd1b67c6f12657426b67331646d1c56ae85b985c5791ba21337f66a6a4c46733"}
Oct 30 12:46:44.750591 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.750519    5837 scope.go:122] "RemoveContainer" containerID="f207bd6d7df7a05e7a26d5d8e36f7ab5fef851e0d640adecb3dbfe6e1d1d7b7a"
Oct 30 12:46:44.760069 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.760045    5837 scope.go:122] "RemoveContainer" containerID="437c97d3095c2d56da056c948fb1374790617c10936928207b5807b1946bde5f"
Oct 30 12:46:44.769490 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.769471    5837 scope.go:122] "RemoveContainer" containerID="f207bd6d7df7a05e7a26d5d8e36f7ab5fef851e0d640adecb3dbfe6e1d1d7b7a"
Oct 30 12:46:44.769786 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:46:44.769766    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f207bd6d7df7a05e7a26d5d8e36f7ab5fef851e0d640adecb3dbfe6e1d1d7b7a\": container with ID starting with f207bd6d7df7a05e7a26d5d8e36f7ab5fef851e0d640adecb3dbfe6e1d1d7b7a not found: ID does not exist" containerID="f207bd6d7df7a05e7a26d5d8e36f7ab5fef851e0d640adecb3dbfe6e1d1d7b7a"
Oct 30 12:46:44.769882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.769794    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f207bd6d7df7a05e7a26d5d8e36f7ab5fef851e0d640adecb3dbfe6e1d1d7b7a"} err="failed to get container status \"f207bd6d7df7a05e7a26d5d8e36f7ab5fef851e0d640adecb3dbfe6e1d1d7b7a\": rpc error: code = NotFound desc = could not find container \"f207bd6d7df7a05e7a26d5d8e36f7ab5fef851e0d640adecb3dbfe6e1d1d7b7a\": container with ID starting with f207bd6d7df7a05e7a26d5d8e36f7ab5fef851e0d640adecb3dbfe6e1d1d7b7a not found: ID does not exist"
Oct 30 12:46:44.769882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.769809    5837 scope.go:122] "RemoveContainer" containerID="437c97d3095c2d56da056c948fb1374790617c10936928207b5807b1946bde5f"
Oct 30 12:46:44.770085 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:46:44.770066    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437c97d3095c2d56da056c948fb1374790617c10936928207b5807b1946bde5f\": container with ID starting with 437c97d3095c2d56da056c948fb1374790617c10936928207b5807b1946bde5f not found: ID does not exist" containerID="437c97d3095c2d56da056c948fb1374790617c10936928207b5807b1946bde5f"
Oct 30 12:46:44.770150 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.770086    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437c97d3095c2d56da056c948fb1374790617c10936928207b5807b1946bde5f"} err="failed to get container status \"437c97d3095c2d56da056c948fb1374790617c10936928207b5807b1946bde5f\": rpc error: code = NotFound desc = could not find container \"437c97d3095c2d56da056c948fb1374790617c10936928207b5807b1946bde5f\": container with ID starting with 437c97d3095c2d56da056c948fb1374790617c10936928207b5807b1946bde5f not found: ID does not exist"
Oct 30 12:46:44.831601 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.831571    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-token\" (UniqueName: \"kubernetes.io/projected/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-token\") pod \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") "
Oct 30 12:46:44.831767 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.831611    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-workload-certs\") pod \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") "
Oct 30 12:46:44.831767 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.831642    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-workload-socket\") pod \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") "
Oct 30 12:46:44.831767 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.831671    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-kube-api-access-ckfcm\" (UniqueName: \"kubernetes.io/projected/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-kube-api-access-ckfcm\") pod \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") "
Oct 30 12:46:44.831767 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.831691    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istiod-ca-cert\") pod \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") "
Oct 30 12:46:44.831767 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.831720    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-data\") pod \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") "
Oct 30 12:46:44.831767 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.831762    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-envoy\") pod \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") "
Oct 30 12:46:44.832075 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.831828    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-podinfo\") pod \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") "
Oct 30 12:46:44.832075 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.831851    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-credential-socket\") pod \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\" (UID: \"1b3fdd85-e545-41c0-9bc8-24db9fb81cb4\") "
Oct 30 12:46:44.832075 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.831982    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-workload-certs" pod "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" (UID: "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:44.832075 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.831987    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-workload-socket" pod "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" (UID: "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:44.832075 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.832058    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istiod-ca-cert" pod "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" (UID: "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:44.832317 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.832187    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-data" pod "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" (UID: "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:44.832317 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.832251    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-credential-socket" pod "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" (UID: "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:44.832317 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.832306    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:44.832509 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.832322    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:44.832509 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.832337    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:44.832509 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.832350    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:44.834209 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.834183    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-envoy" pod "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" (UID: "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:44.834284 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.834188    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-podinfo" pod "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" (UID: "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:46:44.834486 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.834471    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-token" pod "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" (UID: "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:44.834533 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.834503    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-kube-api-access-ckfcm" pod "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" (UID: "1b3fdd85-e545-41c0-9bc8-24db9fb81cb4"). InnerVolumeSpecName "kube-api-access-ckfcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:44.934018 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.933929    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:44.934018 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.933965    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckfcm\" (UniqueName: \"kubernetes.io/projected/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-kube-api-access-ckfcm\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:44.934018 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.933977    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:44.934018 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.933985    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:44.934018 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:44.933994    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:45.097440 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:45.096362    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"]
Oct 30 12:46:45.103454 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:46:45.103302    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:46:45.106965 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:45.106938    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cdd5b3ae-gw-openshift-default-84fc8fc557-4dk7g"]
Oct 30 12:46:45.532677 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:45.532644    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:45.533723 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:45.533704    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:45.611298 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:45.611266    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" path="/var/lib/kubelet/pods/1b3fdd85-e545-41c0-9bc8-24db9fb81cb4/volumes"
Oct 30 12:46:46.411796 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.411752    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg" podUID="6a9dc715-8ac8-483f-ba94-5427d2e6ed2b" containerName="payload-processing" containerID="cri-o://d4180c6701c974f4c313d137606b658e62a472e391af248080154bd6c8a6e612" gracePeriod=25
Oct 30 12:46:46.430317 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.430274    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s" podUID="478ecd8b-4281-4ec1-9aca-bfd2284a435e" containerName="payload-pre-processing" containerID="cri-o://0c9722f2bb7f5fb33d691e8e1e5a06b53d4be5671f01a15cd6cf94e1d391a32f" gracePeriod=25
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.651946    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"]
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652590    5837 cpu_manager.go:414] "containerMap: removing container" podUID="111eeb2b-43df-4895-a5d5-b101432ac848" containerName="payload-pre-processing"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652606    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111eeb2b-43df-4895-a5d5-b101432ac848" containerName="payload-pre-processing"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652627    5837 cpu_manager.go:414] "containerMap: removing container" podUID="f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b" containerName="revoke-keys"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652634    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b" containerName="revoke-keys"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652671    5837 cpu_manager.go:414] "containerMap: removing container" podUID="834dc833-1f8e-439c-a15d-09bf0f851fe7" containerName="payload-pre-processing"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652679    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="834dc833-1f8e-439c-a15d-09bf0f851fe7" containerName="payload-pre-processing"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652691    5837 cpu_manager.go:414] "containerMap: removing container" podUID="5bfec5e9-47b4-4090-a6ef-69172ebbf69f" containerName="payload-processing"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652699    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfec5e9-47b4-4090-a6ef-69172ebbf69f" containerName="payload-processing"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652739    5837 cpu_manager.go:414] "containerMap: removing container" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerName="istio-proxy"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652747    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerName="istio-proxy"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652765    5837 cpu_manager.go:414] "containerMap: removing container" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerName="istio-proxy"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652772    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerName="istio-proxy"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652874    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="111eeb2b-43df-4895-a5d5-b101432ac848" containerName="payload-pre-processing"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652884    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b" containerName="revoke-keys"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652895    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="834dc833-1f8e-439c-a15d-09bf0f851fe7" containerName="payload-pre-processing"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652901    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfec5e9-47b4-4090-a6ef-69172ebbf69f" containerName="payload-processing"
Oct 30 12:46:46.653907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.652911    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerName="istio-proxy"
Oct 30 12:46:46.656228 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.656204    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"
Oct 30 12:46:46.673430 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.673379    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-105228\""
Oct 30 12:46:46.681710 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.681684    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-105228-dockercfg-sjn4l\""
Oct 30 12:46:46.683435 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.683384    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"]
Oct 30 12:46:46.703814 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.703799    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"
Oct 30 12:46:46.723909 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.723794    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"
Oct 30 12:46:46.761761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.761728    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56eb33e2-cc60-4372-8b87-e369b9c12140-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk\" (UID: \"56eb33e2-cc60-4372-8b87-e369b9c12140\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"
Oct 30 12:46:46.761934 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.761915    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74g64\" (UniqueName: \"kubernetes.io/projected/56eb33e2-cc60-4372-8b87-e369b9c12140-kube-api-access-74g64\") pod \"payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk\" (UID: \"56eb33e2-cc60-4372-8b87-e369b9c12140\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"
Oct 30 12:46:46.772133 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.769226    5837 generic.go:356] "Generic (PLEG): container finished" podID="6a9dc715-8ac8-483f-ba94-5427d2e6ed2b" containerID="d4180c6701c974f4c313d137606b658e62a472e391af248080154bd6c8a6e612" exitCode=0
Oct 30 12:46:46.772133 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.769280    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"
Oct 30 12:46:46.772133 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.769316    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg" event={"ID":"6a9dc715-8ac8-483f-ba94-5427d2e6ed2b","Type":"ContainerDied","Data":"d4180c6701c974f4c313d137606b658e62a472e391af248080154bd6c8a6e612"}
Oct 30 12:46:46.772133 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.769453    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg" event={"ID":"6a9dc715-8ac8-483f-ba94-5427d2e6ed2b","Type":"ContainerDied","Data":"89f1a68052d6b4cb62018a1e13347b391cd05266678b110337b980a601a62335"}
Oct 30 12:46:46.772133 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.769491    5837 scope.go:122] "RemoveContainer" containerID="d4180c6701c974f4c313d137606b658e62a472e391af248080154bd6c8a6e612"
Oct 30 12:46:46.772133 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.771086    5837 generic.go:356] "Generic (PLEG): container finished" podID="478ecd8b-4281-4ec1-9aca-bfd2284a435e" containerID="0c9722f2bb7f5fb33d691e8e1e5a06b53d4be5671f01a15cd6cf94e1d391a32f" exitCode=0
Oct 30 12:46:46.772133 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.771157    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s" event={"ID":"478ecd8b-4281-4ec1-9aca-bfd2284a435e","Type":"ContainerDied","Data":"0c9722f2bb7f5fb33d691e8e1e5a06b53d4be5671f01a15cd6cf94e1d391a32f"}
Oct 30 12:46:46.772133 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.771197    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s" event={"ID":"478ecd8b-4281-4ec1-9aca-bfd2284a435e","Type":"ContainerDied","Data":"6df56c3c192caae70c2115c60ef98f826be8641d8b4fe2284529555bd7717968"}
Oct 30 12:46:46.772133 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.771159    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"
Oct 30 12:46:46.781061 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.781033    5837 scope.go:122] "RemoveContainer" containerID="d4180c6701c974f4c313d137606b658e62a472e391af248080154bd6c8a6e612"
Oct 30 12:46:46.781588 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:46:46.781561    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4180c6701c974f4c313d137606b658e62a472e391af248080154bd6c8a6e612\": container with ID starting with d4180c6701c974f4c313d137606b658e62a472e391af248080154bd6c8a6e612 not found: ID does not exist" containerID="d4180c6701c974f4c313d137606b658e62a472e391af248080154bd6c8a6e612"
Oct 30 12:46:46.781732 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.781603    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4180c6701c974f4c313d137606b658e62a472e391af248080154bd6c8a6e612"} err="failed to get container status \"d4180c6701c974f4c313d137606b658e62a472e391af248080154bd6c8a6e612\": rpc error: code = NotFound desc = could not find container \"d4180c6701c974f4c313d137606b658e62a472e391af248080154bd6c8a6e612\": container with ID starting with d4180c6701c974f4c313d137606b658e62a472e391af248080154bd6c8a6e612 not found: ID does not exist"
Oct 30 12:46:46.781732 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.781629    5837 scope.go:122] "RemoveContainer" containerID="0c9722f2bb7f5fb33d691e8e1e5a06b53d4be5671f01a15cd6cf94e1d391a32f"
Oct 30 12:46:46.792665 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.792644    5837 scope.go:122] "RemoveContainer" containerID="0c9722f2bb7f5fb33d691e8e1e5a06b53d4be5671f01a15cd6cf94e1d391a32f"
Oct 30 12:46:46.792942 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:46:46.792922    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c9722f2bb7f5fb33d691e8e1e5a06b53d4be5671f01a15cd6cf94e1d391a32f\": container with ID starting with 0c9722f2bb7f5fb33d691e8e1e5a06b53d4be5671f01a15cd6cf94e1d391a32f not found: ID does not exist" containerID="0c9722f2bb7f5fb33d691e8e1e5a06b53d4be5671f01a15cd6cf94e1d391a32f"
Oct 30 12:46:46.793010 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.792949    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c9722f2bb7f5fb33d691e8e1e5a06b53d4be5671f01a15cd6cf94e1d391a32f"} err="failed to get container status \"0c9722f2bb7f5fb33d691e8e1e5a06b53d4be5671f01a15cd6cf94e1d391a32f\": rpc error: code = NotFound desc = could not find container \"0c9722f2bb7f5fb33d691e8e1e5a06b53d4be5671f01a15cd6cf94e1d391a32f\": container with ID starting with 0c9722f2bb7f5fb33d691e8e1e5a06b53d4be5671f01a15cd6cf94e1d391a32f not found: ID does not exist"
Oct 30 12:46:46.862652 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.862625    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/478ecd8b-4281-4ec1-9aca-bfd2284a435e-kube-api-access-hc4p9\" (UniqueName: \"kubernetes.io/projected/478ecd8b-4281-4ec1-9aca-bfd2284a435e-kube-api-access-hc4p9\") pod \"478ecd8b-4281-4ec1-9aca-bfd2284a435e\" (UID: \"478ecd8b-4281-4ec1-9aca-bfd2284a435e\") "
Oct 30 12:46:46.862792 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.862673    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a9dc715-8ac8-483f-ba94-5427d2e6ed2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9dc715-8ac8-483f-ba94-5427d2e6ed2b-plugins-config-volume\") pod \"6a9dc715-8ac8-483f-ba94-5427d2e6ed2b\" (UID: \"6a9dc715-8ac8-483f-ba94-5427d2e6ed2b\") "
Oct 30 12:46:46.862792 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.862756    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9dc715-8ac8-483f-ba94-5427d2e6ed2b-kube-api-access-92fsb\" (UniqueName: \"kubernetes.io/projected/6a9dc715-8ac8-483f-ba94-5427d2e6ed2b-kube-api-access-92fsb\") pod \"6a9dc715-8ac8-483f-ba94-5427d2e6ed2b\" (UID: \"6a9dc715-8ac8-483f-ba94-5427d2e6ed2b\") "
Oct 30 12:46:46.862920 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.862809    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/478ecd8b-4281-4ec1-9aca-bfd2284a435e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478ecd8b-4281-4ec1-9aca-bfd2284a435e-plugins-config-volume\") pod \"478ecd8b-4281-4ec1-9aca-bfd2284a435e\" (UID: \"478ecd8b-4281-4ec1-9aca-bfd2284a435e\") "
Oct 30 12:46:46.863124 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.863100    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74g64\" (UniqueName: \"kubernetes.io/projected/56eb33e2-cc60-4372-8b87-e369b9c12140-kube-api-access-74g64\") pod \"payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk\" (UID: \"56eb33e2-cc60-4372-8b87-e369b9c12140\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"
Oct 30 12:46:46.863224 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.863203    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56eb33e2-cc60-4372-8b87-e369b9c12140-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk\" (UID: \"56eb33e2-cc60-4372-8b87-e369b9c12140\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"
Oct 30 12:46:46.863358 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.863200    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9dc715-8ac8-483f-ba94-5427d2e6ed2b-plugins-config-volume" pod "6a9dc715-8ac8-483f-ba94-5427d2e6ed2b" (UID: "6a9dc715-8ac8-483f-ba94-5427d2e6ed2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:46.863450 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.863300    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/478ecd8b-4281-4ec1-9aca-bfd2284a435e-plugins-config-volume" pod "478ecd8b-4281-4ec1-9aca-bfd2284a435e" (UID: "478ecd8b-4281-4ec1-9aca-bfd2284a435e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:46.863888 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.863869    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56eb33e2-cc60-4372-8b87-e369b9c12140-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk\" (UID: \"56eb33e2-cc60-4372-8b87-e369b9c12140\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"
Oct 30 12:46:46.864946 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.864926    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478ecd8b-4281-4ec1-9aca-bfd2284a435e-kube-api-access-hc4p9" pod "478ecd8b-4281-4ec1-9aca-bfd2284a435e" (UID: "478ecd8b-4281-4ec1-9aca-bfd2284a435e"). InnerVolumeSpecName "kube-api-access-hc4p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:46.865044 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.865024    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9dc715-8ac8-483f-ba94-5427d2e6ed2b-kube-api-access-92fsb" pod "6a9dc715-8ac8-483f-ba94-5427d2e6ed2b" (UID: "6a9dc715-8ac8-483f-ba94-5427d2e6ed2b"). InnerVolumeSpecName "kube-api-access-92fsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:46.881636 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.881608    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74g64\" (UniqueName: \"kubernetes.io/projected/56eb33e2-cc60-4372-8b87-e369b9c12140-kube-api-access-74g64\") pod \"payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk\" (UID: \"56eb33e2-cc60-4372-8b87-e369b9c12140\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"
Oct 30 12:46:46.964060 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.963979    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478ecd8b-4281-4ec1-9aca-bfd2284a435e-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:46.964060 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.964009    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc4p9\" (UniqueName: \"kubernetes.io/projected/478ecd8b-4281-4ec1-9aca-bfd2284a435e-kube-api-access-hc4p9\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:46.964060 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.964018    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9dc715-8ac8-483f-ba94-5427d2e6ed2b-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:46.964060 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:46.964027    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92fsb\" (UniqueName: \"kubernetes.io/projected/6a9dc715-8ac8-483f-ba94-5427d2e6ed2b-kube-api-access-92fsb\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:47.002381 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:47.002357    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"
Oct 30 12:46:47.165423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:47.165376    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"]
Oct 30 12:46:47.176774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:47.176748    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1b2ed0a-c9fd566bb-9hvvg"]
Oct 30 12:46:47.219002 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:47.218926    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"]
Oct 30 12:46:47.227375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:47.227334    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1b2ed0a-65b5f84df5-8879s"]
Oct 30 12:46:47.295958 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:47.295931    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"]
Oct 30 12:46:47.297346 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:47.297311    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56eb33e2_cc60_4372_8b87_e369b9c12140.slice/crio-95721000284dd9ee3010aa91b54d832e36e1e662c55381d2a7a58f646ce37c81 WatchSource:0}: Error finding container 95721000284dd9ee3010aa91b54d832e36e1e662c55381d2a7a58f646ce37c81: Status 404 returned error can't find the container with id 95721000284dd9ee3010aa91b54d832e36e1e662c55381d2a7a58f646ce37c81
Oct 30 12:46:47.605147 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:47.605112    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="478ecd8b-4281-4ec1-9aca-bfd2284a435e" path="/var/lib/kubelet/pods/478ecd8b-4281-4ec1-9aca-bfd2284a435e/volumes"
Oct 30 12:46:47.605620 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:47.605599    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a9dc715-8ac8-483f-ba94-5427d2e6ed2b" path="/var/lib/kubelet/pods/6a9dc715-8ac8-483f-ba94-5427d2e6ed2b/volumes"
Oct 30 12:46:47.779466 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:47.779430    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk" event={"ID":"56eb33e2-cc60-4372-8b87-e369b9c12140","Type":"ContainerStarted","Data":"f275cd314954c7ce0d23fe70c38ad6bd00164a00b340349eebd7a3ee40ad2d1a"}
Oct 30 12:46:47.779961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:47.779492    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk" event={"ID":"56eb33e2-cc60-4372-8b87-e369b9c12140","Type":"ContainerStarted","Data":"95721000284dd9ee3010aa91b54d832e36e1e662c55381d2a7a58f646ce37c81"}
Oct 30 12:46:47.779961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:47.779523    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"
Oct 30 12:46:47.820690 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:47.820634    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk" podStartSLOduration=1.820620401 podStartE2EDuration="1.820620401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:47.818257622 +0000 UTC m=+596.342541840" watchObservedRunningTime="2026-10-30 12:46:47.820620401 +0000 UTC m=+596.344904619"
Oct 30 12:46:49.161066 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:49.161035    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"]
Oct 30 12:46:50.714721 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:50.714690    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"]
Oct 30 12:46:50.715160 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:50.714949    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7" podUID="e42e36f7-20c2-4259-99c0-a221e144d330" containerName="istio-proxy" containerID="cri-o://c179fad227e65b4b4831f1e6675675095fd26b9bdcbb703925006215a02b6e6c" gracePeriod=30
Oct 30 12:46:50.925078 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:50.925043    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"]
Oct 30 12:46:50.928050 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:50.928020    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48623dd7-9fg6n"]
Oct 30 12:46:51.072419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.072370    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"]
Oct 30 12:46:51.072642 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.072614    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd" podUID="147b05ae-0fc7-49bd-8521-8a092279d46c" containerName="istio-proxy" containerID="cri-o://79f1e5902772b25f8f96fa0c03cdf24ee7974e43f40ba1600dc8b404f3227231" gracePeriod=30
Oct 30 12:46:51.606746 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.606706    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b" path="/var/lib/kubelet/pods/f1b6d32a-6c5b-4610-8d4b-8659cf2bb49b/volumes"
Oct 30 12:46:51.689814 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.689789    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 12:46:51.707249 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.707209    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 12:46:51.837639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.837608    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"]
Oct 30 12:46:51.838449 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.838428    5837 cpu_manager.go:414] "containerMap: removing container" podUID="478ecd8b-4281-4ec1-9aca-bfd2284a435e" containerName="payload-pre-processing"
Oct 30 12:46:51.838502 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.838452    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="478ecd8b-4281-4ec1-9aca-bfd2284a435e" containerName="payload-pre-processing"
Oct 30 12:46:51.838502 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.838483    5837 cpu_manager.go:414] "containerMap: removing container" podUID="6a9dc715-8ac8-483f-ba94-5427d2e6ed2b" containerName="payload-processing"
Oct 30 12:46:51.838502 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.838492    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9dc715-8ac8-483f-ba94-5427d2e6ed2b" containerName="payload-processing"
Oct 30 12:46:51.838616 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.838604    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="478ecd8b-4281-4ec1-9aca-bfd2284a435e" containerName="payload-pre-processing"
Oct 30 12:46:51.838646 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.838621    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9dc715-8ac8-483f-ba94-5427d2e6ed2b" containerName="payload-processing"
Oct 30 12:46:51.838646 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.838632    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3fdd85-e545-41c0-9bc8-24db9fb81cb4" containerName="istio-proxy"
Oct 30 12:46:51.843907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.843886    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"
Oct 30 12:46:51.858364 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.857437    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"]
Oct 30 12:46:51.867476 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.867450    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-53dcb1\""
Oct 30 12:46:51.880141 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.880109    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-53dcb1-dockercfg-7g54z\""
Oct 30 12:46:51.913460 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.913430    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee36af90-164d-404c-be97-5dbaeafc0648-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9\" (UID: \"ee36af90-164d-404c-be97-5dbaeafc0648\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"
Oct 30 12:46:51.913651 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.913487    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtv2b\" (UniqueName: \"kubernetes.io/projected/ee36af90-164d-404c-be97-5dbaeafc0648-kube-api-access-jtv2b\") pod \"payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9\" (UID: \"ee36af90-164d-404c-be97-5dbaeafc0648\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"
Oct 30 12:46:51.934437 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.934373    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"]
Oct 30 12:46:51.939149 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.939125    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"
Oct 30 12:46:51.983044 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:51.983007    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"]
Oct 30 12:46:52.014205 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.014174    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw5dj\" (UniqueName: \"kubernetes.io/projected/bc1998d0-b85e-46aa-a6d4-92b828b66d08-kube-api-access-gw5dj\") pod \"payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q\" (UID: \"bc1998d0-b85e-46aa-a6d4-92b828b66d08\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"
Oct 30 12:46:52.014375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.014226    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee36af90-164d-404c-be97-5dbaeafc0648-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9\" (UID: \"ee36af90-164d-404c-be97-5dbaeafc0648\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"
Oct 30 12:46:52.014477 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.014428    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtv2b\" (UniqueName: \"kubernetes.io/projected/ee36af90-164d-404c-be97-5dbaeafc0648-kube-api-access-jtv2b\") pod \"payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9\" (UID: \"ee36af90-164d-404c-be97-5dbaeafc0648\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"
Oct 30 12:46:52.014543 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.014488    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1998d0-b85e-46aa-a6d4-92b828b66d08-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q\" (UID: \"bc1998d0-b85e-46aa-a6d4-92b828b66d08\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"
Oct 30 12:46:52.015029 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.015004    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee36af90-164d-404c-be97-5dbaeafc0648-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9\" (UID: \"ee36af90-164d-404c-be97-5dbaeafc0648\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"
Oct 30 12:46:52.041869 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.041837    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtv2b\" (UniqueName: \"kubernetes.io/projected/ee36af90-164d-404c-be97-5dbaeafc0648-kube-api-access-jtv2b\") pod \"payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9\" (UID: \"ee36af90-164d-404c-be97-5dbaeafc0648\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"
Oct 30 12:46:52.117470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.115174    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1998d0-b85e-46aa-a6d4-92b828b66d08-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q\" (UID: \"bc1998d0-b85e-46aa-a6d4-92b828b66d08\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"
Oct 30 12:46:52.117470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.115349    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5dj\" (UniqueName: \"kubernetes.io/projected/bc1998d0-b85e-46aa-a6d4-92b828b66d08-kube-api-access-gw5dj\") pod \"payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q\" (UID: \"bc1998d0-b85e-46aa-a6d4-92b828b66d08\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"
Oct 30 12:46:52.117470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.116212    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1998d0-b85e-46aa-a6d4-92b828b66d08-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q\" (UID: \"bc1998d0-b85e-46aa-a6d4-92b828b66d08\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"
Oct 30 12:46:52.138091 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.138060    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5dj\" (UniqueName: \"kubernetes.io/projected/bc1998d0-b85e-46aa-a6d4-92b828b66d08-kube-api-access-gw5dj\") pod \"payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q\" (UID: \"bc1998d0-b85e-46aa-a6d4-92b828b66d08\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"
Oct 30 12:46:52.158012 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.157982    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"
Oct 30 12:46:52.258544 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.258513    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"
Oct 30 12:46:52.357522 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.357480    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"]
Oct 30 12:46:52.362678 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.362652    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.380950 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.380925    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-7a590a-openshift-default-dockercfg-4zgs6\""
Oct 30 12:46:52.382187 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.382164    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"]
Oct 30 12:46:52.455660 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.455343    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"]
Oct 30 12:46:52.520673 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.520645    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.520813 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.520762    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.520891 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.520844    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.520985 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.520909    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.521033 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.521009    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25644850-37a6-4820-8ed2-4f00fedd8274-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.521069 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.521034    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25644850-37a6-4820-8ed2-4f00fedd8274-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.521069 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.521059    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25644850-37a6-4820-8ed2-4f00fedd8274-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.521236 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.521191    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.521310 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.521249    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl89x\" (UniqueName: \"kubernetes.io/projected/25644850-37a6-4820-8ed2-4f00fedd8274-kube-api-access-rl89x\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.550348 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.550327    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"]
Oct 30 12:46:52.622414 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.622336    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.622414 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.622376    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.622613 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.622444    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.622613 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.622492    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25644850-37a6-4820-8ed2-4f00fedd8274-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.622613 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.622521    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25644850-37a6-4820-8ed2-4f00fedd8274-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.622613 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.622558    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25644850-37a6-4820-8ed2-4f00fedd8274-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.622613 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.622609    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.622855 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.622635    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl89x\" (UniqueName: \"kubernetes.io/projected/25644850-37a6-4820-8ed2-4f00fedd8274-kube-api-access-rl89x\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.622855 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.622755    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.622855 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.622819    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.623110 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.623091    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.623514 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.623453    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.623714 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.623673    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25644850-37a6-4820-8ed2-4f00fedd8274-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.623796 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.623729    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.625051 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.625028    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.625456 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.625442    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25644850-37a6-4820-8ed2-4f00fedd8274-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.642768 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.642742    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25644850-37a6-4820-8ed2-4f00fedd8274-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.643253 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.643236    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl89x\" (UniqueName: \"kubernetes.io/projected/25644850-37a6-4820-8ed2-4f00fedd8274-kube-api-access-rl89x\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.679186 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.679153    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:52.691500 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.691474    5837 patch_prober.go:29] interesting pod/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:46:52.691614 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.691523    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7" podUID="e42e36f7-20c2-4259-99c0-a221e144d330" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:46:52.814823 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.814785    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q" event={"ID":"bc1998d0-b85e-46aa-a6d4-92b828b66d08","Type":"ContainerStarted","Data":"ac7b1dbb3c16a1bfe80cde0e1b6a64c91a979b62fb7df37c5173b036c2fee1db"}
Oct 30 12:46:52.815004 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.814851    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"
Oct 30 12:46:52.815004 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.814876    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q" event={"ID":"bc1998d0-b85e-46aa-a6d4-92b828b66d08","Type":"ContainerStarted","Data":"a6733e25188a844d147448cfeeb628127e136c6036346a00925503fb73c5f4d4"}
Oct 30 12:46:52.816543 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.816517    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9" event={"ID":"ee36af90-164d-404c-be97-5dbaeafc0648","Type":"ContainerStarted","Data":"155294b2dc20a7b61ec74a3be78fd3cdbd112919afc821e8ff008f839b3276b8"}
Oct 30 12:46:52.816643 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.816557    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9" event={"ID":"ee36af90-164d-404c-be97-5dbaeafc0648","Type":"ContainerStarted","Data":"27a430444a3b657f32dedff80b6123fc53523d34c4bae32b939cdf6b783a2ad5"}
Oct 30 12:46:52.816701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.816662    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"
Oct 30 12:46:52.855034 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.854985    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q" podStartSLOduration=1.854959099 podStartE2EDuration="1.854959099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:52.853519735 +0000 UTC m=+601.377803954" watchObservedRunningTime="2026-10-30 12:46:52.854959099 +0000 UTC m=+601.379243321"
Oct 30 12:46:52.890934 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.890854    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9" podStartSLOduration=1.8908425819999999 podStartE2EDuration="1.890842582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:52.889543414 +0000 UTC m=+601.413827632" watchObservedRunningTime="2026-10-30 12:46:52.890842582 +0000 UTC m=+601.415126801"
Oct 30 12:46:52.986777 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.986751    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"]
Oct 30 12:46:52.989698 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.989669    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:52.989805 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.989734    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:52.989805 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:52.989762    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:46:53.824184 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:53.824150    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" event={"ID":"25644850-37a6-4820-8ed2-4f00fedd8274","Type":"ContainerStarted","Data":"23974a71b9810c8bd9b28fb891a606cff350e664ba3fda6e41453f8ed2a22c8a"}
Oct 30 12:46:53.824343 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:53.824210    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" event={"ID":"25644850-37a6-4820-8ed2-4f00fedd8274","Type":"ContainerStarted","Data":"530f85b8be1110cd90cca74f0f1e7762b9b92f1f964f6ced9b386430d40c6ef5"}
Oct 30 12:46:53.865670 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:53.865610    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" podStartSLOduration=1.865593047 podStartE2EDuration="1.865593047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:53.86394083 +0000 UTC m=+602.388225050" watchObservedRunningTime="2026-10-30 12:46:53.865593047 +0000 UTC m=+602.389877268"
Oct 30 12:46:54.161708 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.161613    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm" podUID="1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d" containerName="payload-processing" containerID="cri-o://c84bc09853ccb112268ceb1a9ac22c3aac04a7ecfabeeea41655fb83bcd1ebf4" gracePeriod=25
Oct 30 12:46:54.414284 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.414232    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"
Oct 30 12:46:54.441630 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.441605    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d-plugins-config-volume\") pod \"1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d\" (UID: \"1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d\") "
Oct 30 12:46:54.441745 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.441663    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d-kube-api-access-s78ww\" (UniqueName: \"kubernetes.io/projected/1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d-kube-api-access-s78ww\") pod \"1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d\" (UID: \"1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d\") "
Oct 30 12:46:54.442002 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.441980    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d-plugins-config-volume" pod "1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d" (UID: "1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:54.443792 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.443773    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d-kube-api-access-s78ww" pod "1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d" (UID: "1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d"). InnerVolumeSpecName "kube-api-access-s78ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:54.543146 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.543105    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s78ww\" (UniqueName: \"kubernetes.io/projected/1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d-kube-api-access-s78ww\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:54.543146 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.543140    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:54.679889 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.679813    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:54.684731 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.684706    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:54.831647 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.831615    5837 generic.go:356] "Generic (PLEG): container finished" podID="1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d" containerID="c84bc09853ccb112268ceb1a9ac22c3aac04a7ecfabeeea41655fb83bcd1ebf4" exitCode=0
Oct 30 12:46:54.831832 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.831666    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"
Oct 30 12:46:54.831832 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.831702    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm" event={"ID":"1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d","Type":"ContainerDied","Data":"c84bc09853ccb112268ceb1a9ac22c3aac04a7ecfabeeea41655fb83bcd1ebf4"}
Oct 30 12:46:54.831832 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.831755    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm" event={"ID":"1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d","Type":"ContainerDied","Data":"3ec80b07b27501f75418d529b7b053923ce19497bc4e1199871994375ea1d8ba"}
Oct 30 12:46:54.831832 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.831780    5837 scope.go:122] "RemoveContainer" containerID="c84bc09853ccb112268ceb1a9ac22c3aac04a7ecfabeeea41655fb83bcd1ebf4"
Oct 30 12:46:54.832103 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.832087    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:54.833352 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.833329    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:46:54.841308 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.841293    5837 scope.go:122] "RemoveContainer" containerID="c84bc09853ccb112268ceb1a9ac22c3aac04a7ecfabeeea41655fb83bcd1ebf4"
Oct 30 12:46:54.841604 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:46:54.841587    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c84bc09853ccb112268ceb1a9ac22c3aac04a7ecfabeeea41655fb83bcd1ebf4\": container with ID starting with c84bc09853ccb112268ceb1a9ac22c3aac04a7ecfabeeea41655fb83bcd1ebf4 not found: ID does not exist" containerID="c84bc09853ccb112268ceb1a9ac22c3aac04a7ecfabeeea41655fb83bcd1ebf4"
Oct 30 12:46:54.841660 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.841614    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c84bc09853ccb112268ceb1a9ac22c3aac04a7ecfabeeea41655fb83bcd1ebf4"} err="failed to get container status \"c84bc09853ccb112268ceb1a9ac22c3aac04a7ecfabeeea41655fb83bcd1ebf4\": rpc error: code = NotFound desc = could not find container \"c84bc09853ccb112268ceb1a9ac22c3aac04a7ecfabeeea41655fb83bcd1ebf4\": container with ID starting with c84bc09853ccb112268ceb1a9ac22c3aac04a7ecfabeeea41655fb83bcd1ebf4 not found: ID does not exist"
Oct 30 12:46:54.882361 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.882328    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"]
Oct 30 12:46:54.885653 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:54.885626    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-48623dd7-58d567b984-j9qxm"]
Oct 30 12:46:55.604881 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:55.604846    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d" path="/var/lib/kubelet/pods/1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d/volumes"
Oct 30 12:46:55.842369 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:55.842341    5837 generic.go:356] "Generic (PLEG): container finished" podID="e42e36f7-20c2-4259-99c0-a221e144d330" containerID="c179fad227e65b4b4831f1e6675675095fd26b9bdcbb703925006215a02b6e6c" exitCode=0
Oct 30 12:46:55.842601 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:55.842461    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7" event={"ID":"e42e36f7-20c2-4259-99c0-a221e144d330","Type":"ContainerDied","Data":"c179fad227e65b4b4831f1e6675675095fd26b9bdcbb703925006215a02b6e6c"}
Oct 30 12:46:55.842601 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:55.842530    5837 scope.go:122] "RemoveContainer" containerID="df466c4ae60f81ed662c1b8738a26830e68a8702e3e6ba31b330ed4df0b255b9"
Oct 30 12:46:55.970378 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:55.970354    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:56.058949 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.058914    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e42e36f7-20c2-4259-99c0-a221e144d330-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42e36f7-20c2-4259-99c0-a221e144d330-istio-podinfo\") pod \"e42e36f7-20c2-4259-99c0-a221e144d330\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") "
Oct 30 12:46:56.059116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.058995    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-credential-socket\") pod \"e42e36f7-20c2-4259-99c0-a221e144d330\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") "
Oct 30 12:46:56.059116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059046    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-workload-certs\") pod \"e42e36f7-20c2-4259-99c0-a221e144d330\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") "
Oct 30 12:46:56.059116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059082    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-istio-data\") pod \"e42e36f7-20c2-4259-99c0-a221e144d330\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") "
Oct 30 12:46:56.059116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059105    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-workload-socket\") pod \"e42e36f7-20c2-4259-99c0-a221e144d330\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") "
Oct 30 12:46:56.059343 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059131    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-istio-envoy\") pod \"e42e36f7-20c2-4259-99c0-a221e144d330\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") "
Oct 30 12:46:56.059343 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059161    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e42e36f7-20c2-4259-99c0-a221e144d330-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42e36f7-20c2-4259-99c0-a221e144d330-istiod-ca-cert\") pod \"e42e36f7-20c2-4259-99c0-a221e144d330\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") "
Oct 30 12:46:56.059343 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059194    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42e36f7-20c2-4259-99c0-a221e144d330-kube-api-access-btv5p\" (UniqueName: \"kubernetes.io/projected/e42e36f7-20c2-4259-99c0-a221e144d330-kube-api-access-btv5p\") pod \"e42e36f7-20c2-4259-99c0-a221e144d330\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") "
Oct 30 12:46:56.059343 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059222    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42e36f7-20c2-4259-99c0-a221e144d330-istio-token\" (UniqueName: \"kubernetes.io/projected/e42e36f7-20c2-4259-99c0-a221e144d330-istio-token\") pod \"e42e36f7-20c2-4259-99c0-a221e144d330\" (UID: \"e42e36f7-20c2-4259-99c0-a221e144d330\") "
Oct 30 12:46:56.059605 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059334    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-credential-socket" pod "e42e36f7-20c2-4259-99c0-a221e144d330" (UID: "e42e36f7-20c2-4259-99c0-a221e144d330"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:56.059605 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059519    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-workload-certs" pod "e42e36f7-20c2-4259-99c0-a221e144d330" (UID: "e42e36f7-20c2-4259-99c0-a221e144d330"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:56.059605 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059539    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-istio-data" pod "e42e36f7-20c2-4259-99c0-a221e144d330" (UID: "e42e36f7-20c2-4259-99c0-a221e144d330"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:56.059741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059622    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42e36f7-20c2-4259-99c0-a221e144d330-istiod-ca-cert" pod "e42e36f7-20c2-4259-99c0-a221e144d330" (UID: "e42e36f7-20c2-4259-99c0-a221e144d330"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:56.059741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059622    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-workload-socket" pod "e42e36f7-20c2-4259-99c0-a221e144d330" (UID: "e42e36f7-20c2-4259-99c0-a221e144d330"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:56.060009 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059794    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.060009 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059821    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.060009 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059840    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.060009 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059854    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.060009 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.059867    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42e36f7-20c2-4259-99c0-a221e144d330-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.061829 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.061780    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e42e36f7-20c2-4259-99c0-a221e144d330-istio-podinfo" pod "e42e36f7-20c2-4259-99c0-a221e144d330" (UID: "e42e36f7-20c2-4259-99c0-a221e144d330"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:46:56.061829 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.061815    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42e36f7-20c2-4259-99c0-a221e144d330-kube-api-access-btv5p" pod "e42e36f7-20c2-4259-99c0-a221e144d330" (UID: "e42e36f7-20c2-4259-99c0-a221e144d330"). InnerVolumeSpecName "kube-api-access-btv5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:56.062034 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.061833    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-istio-envoy" pod "e42e36f7-20c2-4259-99c0-a221e144d330" (UID: "e42e36f7-20c2-4259-99c0-a221e144d330"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:56.062034 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.061871    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42e36f7-20c2-4259-99c0-a221e144d330-istio-token" pod "e42e36f7-20c2-4259-99c0-a221e144d330" (UID: "e42e36f7-20c2-4259-99c0-a221e144d330"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:56.161353 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.161274    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42e36f7-20c2-4259-99c0-a221e144d330-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.161353 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.161302    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btv5p\" (UniqueName: \"kubernetes.io/projected/e42e36f7-20c2-4259-99c0-a221e144d330-kube-api-access-btv5p\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.161353 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.161312    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42e36f7-20c2-4259-99c0-a221e144d330-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.161353 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.161322    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42e36f7-20c2-4259-99c0-a221e144d330-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.229580 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:46:56.229531    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147b05ae_0fc7_49bd_8521_8a092279d46c.slice/crio-conmon-79f1e5902772b25f8f96fa0c03cdf24ee7974e43f40ba1600dc8b404f3227231.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42e36f7_20c2_4259_99c0_a221e144d330.slice/crio-c179fad227e65b4b4831f1e6675675095fd26b9bdcbb703925006215a02b6e6c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7adf61_0dc7_4e45_ae9e_86f6cd1e8f9d.slice/crio-c84bc09853ccb112268ceb1a9ac22c3aac04a7ecfabeeea41655fb83bcd1ebf4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147b05ae_0fc7_49bd_8521_8a092279d46c.slice/crio-79f1e5902772b25f8f96fa0c03cdf24ee7974e43f40ba1600dc8b404f3227231.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7adf61_0dc7_4e45_ae9e_86f6cd1e8f9d.slice/crio-3ec80b07b27501f75418d529b7b053923ce19497bc4e1199871994375ea1d8ba\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7adf61_0dc7_4e45_ae9e_86f6cd1e8f9d.slice/crio-conmon-c84bc09853ccb112268ceb1a9ac22c3aac04a7ecfabeeea41655fb83bcd1ebf4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42e36f7_20c2_4259_99c0_a221e144d330.slice/crio-conmon-c179fad227e65b4b4831f1e6675675095fd26b9bdcbb703925006215a02b6e6c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7adf61_0dc7_4e45_ae9e_86f6cd1e8f9d.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 12:46:56.308090 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.308066    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:56.363311 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.363280    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-workload-certs\") pod \"147b05ae-0fc7-49bd-8521-8a092279d46c\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") "
Oct 30 12:46:56.363548 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.363353    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-workload-socket\") pod \"147b05ae-0fc7-49bd-8521-8a092279d46c\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") "
Oct 30 12:46:56.363548 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.363420    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-data\") pod \"147b05ae-0fc7-49bd-8521-8a092279d46c\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") "
Oct 30 12:46:56.363548 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.363469    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147b05ae-0fc7-49bd-8521-8a092279d46c-kube-api-access-485rm\" (UniqueName: \"kubernetes.io/projected/147b05ae-0fc7-49bd-8521-8a092279d46c-kube-api-access-485rm\") pod \"147b05ae-0fc7-49bd-8521-8a092279d46c\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") "
Oct 30 12:46:56.363548 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.363546    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147b05ae-0fc7-49bd-8521-8a092279d46c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147b05ae-0fc7-49bd-8521-8a092279d46c-istiod-ca-cert\") pod \"147b05ae-0fc7-49bd-8521-8a092279d46c\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") "
Oct 30 12:46:56.363782 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.363559    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-workload-certs" pod "147b05ae-0fc7-49bd-8521-8a092279d46c" (UID: "147b05ae-0fc7-49bd-8521-8a092279d46c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:56.363782 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.363578    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-credential-socket\") pod \"147b05ae-0fc7-49bd-8521-8a092279d46c\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") "
Oct 30 12:46:56.363782 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.363631    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-podinfo\") pod \"147b05ae-0fc7-49bd-8521-8a092279d46c\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") "
Oct 30 12:46:56.363782 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.363726    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-envoy\") pod \"147b05ae-0fc7-49bd-8521-8a092279d46c\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") "
Oct 30 12:46:56.363987 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.363828    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-credential-socket" pod "147b05ae-0fc7-49bd-8521-8a092279d46c" (UID: "147b05ae-0fc7-49bd-8521-8a092279d46c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:56.363987 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.363857    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-token\" (UniqueName: \"kubernetes.io/projected/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-token\") pod \"147b05ae-0fc7-49bd-8521-8a092279d46c\" (UID: \"147b05ae-0fc7-49bd-8521-8a092279d46c\") "
Oct 30 12:46:56.364387 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.364366    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.364572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.364555    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.365218 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.364723    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-workload-socket" pod "147b05ae-0fc7-49bd-8521-8a092279d46c" (UID: "147b05ae-0fc7-49bd-8521-8a092279d46c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:56.365512 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.365295    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147b05ae-0fc7-49bd-8521-8a092279d46c-istiod-ca-cert" pod "147b05ae-0fc7-49bd-8521-8a092279d46c" (UID: "147b05ae-0fc7-49bd-8521-8a092279d46c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:56.365927 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.365898    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-data" pod "147b05ae-0fc7-49bd-8521-8a092279d46c" (UID: "147b05ae-0fc7-49bd-8521-8a092279d46c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:56.366870 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.366844    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-podinfo" pod "147b05ae-0fc7-49bd-8521-8a092279d46c" (UID: "147b05ae-0fc7-49bd-8521-8a092279d46c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:46:56.368163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.368139    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-envoy" pod "147b05ae-0fc7-49bd-8521-8a092279d46c" (UID: "147b05ae-0fc7-49bd-8521-8a092279d46c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:56.368163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.368145    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147b05ae-0fc7-49bd-8521-8a092279d46c-kube-api-access-485rm" pod "147b05ae-0fc7-49bd-8521-8a092279d46c" (UID: "147b05ae-0fc7-49bd-8521-8a092279d46c"). InnerVolumeSpecName "kube-api-access-485rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:56.368612 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.368594    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-token" pod "147b05ae-0fc7-49bd-8521-8a092279d46c" (UID: "147b05ae-0fc7-49bd-8521-8a092279d46c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:56.465421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.465307    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.465421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.465338    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-485rm\" (UniqueName: \"kubernetes.io/projected/147b05ae-0fc7-49bd-8521-8a092279d46c-kube-api-access-485rm\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.465421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.465351    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147b05ae-0fc7-49bd-8521-8a092279d46c-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.465421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.465366    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.465421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.465377    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.465421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.465388    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147b05ae-0fc7-49bd-8521-8a092279d46c-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.465714 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.465430    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b05ae-0fc7-49bd-8521-8a092279d46c-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:46:56.850315 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.850284    5837 generic.go:356] "Generic (PLEG): container finished" podID="147b05ae-0fc7-49bd-8521-8a092279d46c" containerID="79f1e5902772b25f8f96fa0c03cdf24ee7974e43f40ba1600dc8b404f3227231" exitCode=0
Oct 30 12:46:56.850514 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.850334    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"
Oct 30 12:46:56.850514 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.850374    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd" event={"ID":"147b05ae-0fc7-49bd-8521-8a092279d46c","Type":"ContainerDied","Data":"79f1e5902772b25f8f96fa0c03cdf24ee7974e43f40ba1600dc8b404f3227231"}
Oct 30 12:46:56.850514 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.850458    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd" event={"ID":"147b05ae-0fc7-49bd-8521-8a092279d46c","Type":"ContainerDied","Data":"a47627496cbb13263279ec2b8b621a601e742f0d7e13b58ad8459466e46c4065"}
Oct 30 12:46:56.850514 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.850483    5837 scope.go:122] "RemoveContainer" containerID="79f1e5902772b25f8f96fa0c03cdf24ee7974e43f40ba1600dc8b404f3227231"
Oct 30 12:46:56.852193 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.852162    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7" event={"ID":"e42e36f7-20c2-4259-99c0-a221e144d330","Type":"ContainerDied","Data":"fdd786d48c7a7d99a79fc6fa7d6f7033190afc26eeb07c1b61520306eac71f07"}
Oct 30 12:46:56.852366 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.852350    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"
Oct 30 12:46:56.862216 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.862191    5837 scope.go:122] "RemoveContainer" containerID="ef05778be2425a7cab00379c1d0898b7eb34f711a9e65a183cdc89a4479d8315"
Oct 30 12:46:56.873211 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.873193    5837 scope.go:122] "RemoveContainer" containerID="79f1e5902772b25f8f96fa0c03cdf24ee7974e43f40ba1600dc8b404f3227231"
Oct 30 12:46:56.873522 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:46:56.873503    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f1e5902772b25f8f96fa0c03cdf24ee7974e43f40ba1600dc8b404f3227231\": container with ID starting with 79f1e5902772b25f8f96fa0c03cdf24ee7974e43f40ba1600dc8b404f3227231 not found: ID does not exist" containerID="79f1e5902772b25f8f96fa0c03cdf24ee7974e43f40ba1600dc8b404f3227231"
Oct 30 12:46:56.873583 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.873538    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f1e5902772b25f8f96fa0c03cdf24ee7974e43f40ba1600dc8b404f3227231"} err="failed to get container status \"79f1e5902772b25f8f96fa0c03cdf24ee7974e43f40ba1600dc8b404f3227231\": rpc error: code = NotFound desc = could not find container \"79f1e5902772b25f8f96fa0c03cdf24ee7974e43f40ba1600dc8b404f3227231\": container with ID starting with 79f1e5902772b25f8f96fa0c03cdf24ee7974e43f40ba1600dc8b404f3227231 not found: ID does not exist"
Oct 30 12:46:56.873583 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.873556    5837 scope.go:122] "RemoveContainer" containerID="ef05778be2425a7cab00379c1d0898b7eb34f711a9e65a183cdc89a4479d8315"
Oct 30 12:46:56.873816 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:46:56.873798    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef05778be2425a7cab00379c1d0898b7eb34f711a9e65a183cdc89a4479d8315\": container with ID starting with ef05778be2425a7cab00379c1d0898b7eb34f711a9e65a183cdc89a4479d8315 not found: ID does not exist" containerID="ef05778be2425a7cab00379c1d0898b7eb34f711a9e65a183cdc89a4479d8315"
Oct 30 12:46:56.873883 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.873830    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef05778be2425a7cab00379c1d0898b7eb34f711a9e65a183cdc89a4479d8315"} err="failed to get container status \"ef05778be2425a7cab00379c1d0898b7eb34f711a9e65a183cdc89a4479d8315\": rpc error: code = NotFound desc = could not find container \"ef05778be2425a7cab00379c1d0898b7eb34f711a9e65a183cdc89a4479d8315\": container with ID starting with ef05778be2425a7cab00379c1d0898b7eb34f711a9e65a183cdc89a4479d8315 not found: ID does not exist"
Oct 30 12:46:56.873883 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.873852    5837 scope.go:122] "RemoveContainer" containerID="c179fad227e65b4b4831f1e6675675095fd26b9bdcbb703925006215a02b6e6c"
Oct 30 12:46:56.921369 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.921336    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"]
Oct 30 12:46:56.932807 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.932774    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-48623dd7-openshift-default-5c8b5957fc-w55zd"]
Oct 30 12:46:56.971715 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.971680    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"]
Oct 30 12:46:56.982179 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:56.982142    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5431fa34-openshift-default-6685769f9-p95b7"]
Oct 30 12:46:57.103467 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.103305    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"]
Oct 30 12:46:57.107415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.104054    5837 cpu_manager.go:414] "containerMap: removing container" podUID="147b05ae-0fc7-49bd-8521-8a092279d46c" containerName="istio-proxy"
Oct 30 12:46:57.107415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.104077    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147b05ae-0fc7-49bd-8521-8a092279d46c" containerName="istio-proxy"
Oct 30 12:46:57.107415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.104124    5837 cpu_manager.go:414] "containerMap: removing container" podUID="147b05ae-0fc7-49bd-8521-8a092279d46c" containerName="istio-proxy"
Oct 30 12:46:57.107415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.104132    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147b05ae-0fc7-49bd-8521-8a092279d46c" containerName="istio-proxy"
Oct 30 12:46:57.107415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.104161    5837 cpu_manager.go:414] "containerMap: removing container" podUID="e42e36f7-20c2-4259-99c0-a221e144d330" containerName="istio-proxy"
Oct 30 12:46:57.107415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.104169    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42e36f7-20c2-4259-99c0-a221e144d330" containerName="istio-proxy"
Oct 30 12:46:57.107415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.104185    5837 cpu_manager.go:414] "containerMap: removing container" podUID="e42e36f7-20c2-4259-99c0-a221e144d330" containerName="istio-proxy"
Oct 30 12:46:57.107415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.104191    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42e36f7-20c2-4259-99c0-a221e144d330" containerName="istio-proxy"
Oct 30 12:46:57.107415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.104201    5837 cpu_manager.go:414] "containerMap: removing container" podUID="1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d" containerName="payload-processing"
Oct 30 12:46:57.107415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.104207    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d" containerName="payload-processing"
Oct 30 12:46:57.107415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.104297    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="147b05ae-0fc7-49bd-8521-8a092279d46c" containerName="istio-proxy"
Oct 30 12:46:57.107415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.104313    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="147b05ae-0fc7-49bd-8521-8a092279d46c" containerName="istio-proxy"
Oct 30 12:46:57.107415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.104328    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42e36f7-20c2-4259-99c0-a221e144d330" containerName="istio-proxy"
Oct 30 12:46:57.107415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.104338    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42e36f7-20c2-4259-99c0-a221e144d330" containerName="istio-proxy"
Oct 30 12:46:57.107415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.104345    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7adf61-0dc7-4e45-ae9e-86f6cd1e8f9d" containerName="payload-processing"
Oct 30 12:46:57.111414 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.110020    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"
Oct 30 12:46:57.130707 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.128133    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-c81f42-dockercfg-hqjwx\""
Oct 30 12:46:57.131019 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.130892    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"]
Oct 30 12:46:57.131699 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.131678    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-c81f42\""
Oct 30 12:46:57.183687 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.183631    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4895db-c87e-45f5-9317-e2643409cff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9\" (UID: \"1c4895db-c87e-45f5-9317-e2643409cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"
Oct 30 12:46:57.183872 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.183765    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxr95\" (UniqueName: \"kubernetes.io/projected/1c4895db-c87e-45f5-9317-e2643409cff3-kube-api-access-cxr95\") pod \"payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9\" (UID: \"1c4895db-c87e-45f5-9317-e2643409cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"
Oct 30 12:46:57.195731 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.195695    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"]
Oct 30 12:46:57.199873 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.199847    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"
Oct 30 12:46:57.227809 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.222419    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"]
Oct 30 12:46:57.285056 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.285015    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4895db-c87e-45f5-9317-e2643409cff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9\" (UID: \"1c4895db-c87e-45f5-9317-e2643409cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"
Oct 30 12:46:57.285230 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.285106    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxr95\" (UniqueName: \"kubernetes.io/projected/1c4895db-c87e-45f5-9317-e2643409cff3-kube-api-access-cxr95\") pod \"payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9\" (UID: \"1c4895db-c87e-45f5-9317-e2643409cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"
Oct 30 12:46:57.285230 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.285136    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrfjp\" (UniqueName: \"kubernetes.io/projected/61c702c3-5d1c-44d9-b02e-1fa7e139ca9c-kube-api-access-jrfjp\") pod \"payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm\" (UID: \"61c702c3-5d1c-44d9-b02e-1fa7e139ca9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"
Oct 30 12:46:57.285230 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.285168    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c702c3-5d1c-44d9-b02e-1fa7e139ca9c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm\" (UID: \"61c702c3-5d1c-44d9-b02e-1fa7e139ca9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"
Oct 30 12:46:57.285974 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.285952    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4895db-c87e-45f5-9317-e2643409cff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9\" (UID: \"1c4895db-c87e-45f5-9317-e2643409cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"
Oct 30 12:46:57.304373 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.304348    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxr95\" (UniqueName: \"kubernetes.io/projected/1c4895db-c87e-45f5-9317-e2643409cff3-kube-api-access-cxr95\") pod \"payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9\" (UID: \"1c4895db-c87e-45f5-9317-e2643409cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"
Oct 30 12:46:57.386465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.386365    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c702c3-5d1c-44d9-b02e-1fa7e139ca9c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm\" (UID: \"61c702c3-5d1c-44d9-b02e-1fa7e139ca9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"
Oct 30 12:46:57.386631 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.386616    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfjp\" (UniqueName: \"kubernetes.io/projected/61c702c3-5d1c-44d9-b02e-1fa7e139ca9c-kube-api-access-jrfjp\") pod \"payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm\" (UID: \"61c702c3-5d1c-44d9-b02e-1fa7e139ca9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"
Oct 30 12:46:57.387306 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.387279    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c702c3-5d1c-44d9-b02e-1fa7e139ca9c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm\" (UID: \"61c702c3-5d1c-44d9-b02e-1fa7e139ca9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"
Oct 30 12:46:57.405842 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.405808    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrfjp\" (UniqueName: \"kubernetes.io/projected/61c702c3-5d1c-44d9-b02e-1fa7e139ca9c-kube-api-access-jrfjp\") pod \"payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm\" (UID: \"61c702c3-5d1c-44d9-b02e-1fa7e139ca9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"
Oct 30 12:46:57.438674 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.438628    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"
Oct 30 12:46:57.518994 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.518753    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"
Oct 30 12:46:57.605537 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.605507    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147b05ae-0fc7-49bd-8521-8a092279d46c" path="/var/lib/kubelet/pods/147b05ae-0fc7-49bd-8521-8a092279d46c/volumes"
Oct 30 12:46:57.605958 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.605945    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e42e36f7-20c2-4259-99c0-a221e144d330" path="/var/lib/kubelet/pods/e42e36f7-20c2-4259-99c0-a221e144d330/volumes"
Oct 30 12:46:57.760754 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.760729    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"]
Oct 30 12:46:57.762862 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:57.762834    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4895db_c87e_45f5_9317_e2643409cff3.slice/crio-2da41c00a0926613afe680d5270b693b700e5847c7d472ebac1c08b531b4ae79 WatchSource:0}: Error finding container 2da41c00a0926613afe680d5270b693b700e5847c7d472ebac1c08b531b4ae79: Status 404 returned error can't find the container with id 2da41c00a0926613afe680d5270b693b700e5847c7d472ebac1c08b531b4ae79
Oct 30 12:46:57.815284 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.814353    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"]
Oct 30 12:46:57.818031 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:57.818006    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c702c3_5d1c_44d9_b02e_1fa7e139ca9c.slice/crio-3abe0a877ac203afa156eb9c5a9bd7e4e4354ee0ba81567bc9528f28c0bed5aa WatchSource:0}: Error finding container 3abe0a877ac203afa156eb9c5a9bd7e4e4354ee0ba81567bc9528f28c0bed5aa: Status 404 returned error can't find the container with id 3abe0a877ac203afa156eb9c5a9bd7e4e4354ee0ba81567bc9528f28c0bed5aa
Oct 30 12:46:57.859809 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.859773    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9" event={"ID":"1c4895db-c87e-45f5-9317-e2643409cff3","Type":"ContainerStarted","Data":"bb428992badd8568939c0307d90c4fa52f70eb08cc8b7541ba6531cbbd7c5963"}
Oct 30 12:46:57.859961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.859840    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"
Oct 30 12:46:57.859961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.859862    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9" event={"ID":"1c4895db-c87e-45f5-9317-e2643409cff3","Type":"ContainerStarted","Data":"2da41c00a0926613afe680d5270b693b700e5847c7d472ebac1c08b531b4ae79"}
Oct 30 12:46:57.862208 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.862185    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm" event={"ID":"61c702c3-5d1c-44d9-b02e-1fa7e139ca9c","Type":"ContainerStarted","Data":"3abe0a877ac203afa156eb9c5a9bd7e4e4354ee0ba81567bc9528f28c0bed5aa"}
Oct 30 12:46:57.898789 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:57.898710    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9" podStartSLOduration=0.898691941 podStartE2EDuration="898.691941ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:57.897064511 +0000 UTC m=+606.421348731" watchObservedRunningTime="2026-10-30 12:46:57.898691941 +0000 UTC m=+606.422976161"
Oct 30 12:46:58.790553 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:58.790526    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"
Oct 30 12:46:58.824314 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:58.824269    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"]
Oct 30 12:46:58.828645 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:58.828619    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"
Oct 30 12:46:58.844501 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:58.844467    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-7a590a\""
Oct 30 12:46:58.852095 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:58.852058    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"]
Oct 30 12:46:58.869319 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:58.869290    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-7a590a-dockercfg-c27sd\""
Oct 30 12:46:58.870935 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:58.870900    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm" event={"ID":"61c702c3-5d1c-44d9-b02e-1fa7e139ca9c","Type":"ContainerStarted","Data":"a49be8fc2aeb78b901f2ac9f4947b7612f09e20680cb493db39ae4a38145ea7a"}
Oct 30 12:46:58.871080 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:58.870982    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"
Oct 30 12:46:58.896287 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:58.896152    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"]
Oct 30 12:46:58.900134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:58.900113    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"
Oct 30 12:46:58.903924 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:58.903900    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mnvk\" (UniqueName: \"kubernetes.io/projected/3929deb3-fa21-4e2f-8ada-6348c9faac3d-kube-api-access-2mnvk\") pod \"payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv\" (UID: \"3929deb3-fa21-4e2f-8ada-6348c9faac3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"
Oct 30 12:46:58.904057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:58.903940    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3929deb3-fa21-4e2f-8ada-6348c9faac3d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv\" (UID: \"3929deb3-fa21-4e2f-8ada-6348c9faac3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"
Oct 30 12:46:58.918994 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:58.918969    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"]
Oct 30 12:46:58.939545 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:58.939471    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm" podStartSLOduration=1.939452661 podStartE2EDuration="1.939452661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:58.937472286 +0000 UTC m=+607.461756524" watchObservedRunningTime="2026-10-30 12:46:58.939452661 +0000 UTC m=+607.463736894"
Oct 30 12:46:59.005513 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.005484    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l982h\" (UniqueName: \"kubernetes.io/projected/fb963087-5281-48f9-b322-110ae1590936-kube-api-access-l982h\") pod \"payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv\" (UID: \"fb963087-5281-48f9-b322-110ae1590936\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"
Oct 30 12:46:59.005681 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.005536    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mnvk\" (UniqueName: \"kubernetes.io/projected/3929deb3-fa21-4e2f-8ada-6348c9faac3d-kube-api-access-2mnvk\") pod \"payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv\" (UID: \"3929deb3-fa21-4e2f-8ada-6348c9faac3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"
Oct 30 12:46:59.005681 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.005636    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3929deb3-fa21-4e2f-8ada-6348c9faac3d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv\" (UID: \"3929deb3-fa21-4e2f-8ada-6348c9faac3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"
Oct 30 12:46:59.005752 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.005729    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb963087-5281-48f9-b322-110ae1590936-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv\" (UID: \"fb963087-5281-48f9-b322-110ae1590936\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"
Oct 30 12:46:59.006323 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.006305    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3929deb3-fa21-4e2f-8ada-6348c9faac3d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv\" (UID: \"3929deb3-fa21-4e2f-8ada-6348c9faac3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"
Oct 30 12:46:59.025224 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.025199    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mnvk\" (UniqueName: \"kubernetes.io/projected/3929deb3-fa21-4e2f-8ada-6348c9faac3d-kube-api-access-2mnvk\") pod \"payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv\" (UID: \"3929deb3-fa21-4e2f-8ada-6348c9faac3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"
Oct 30 12:46:59.106841 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.106741    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l982h\" (UniqueName: \"kubernetes.io/projected/fb963087-5281-48f9-b322-110ae1590936-kube-api-access-l982h\") pod \"payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv\" (UID: \"fb963087-5281-48f9-b322-110ae1590936\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"
Oct 30 12:46:59.107026 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.106856    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb963087-5281-48f9-b322-110ae1590936-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv\" (UID: \"fb963087-5281-48f9-b322-110ae1590936\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"
Oct 30 12:46:59.107765 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.107737    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb963087-5281-48f9-b322-110ae1590936-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv\" (UID: \"fb963087-5281-48f9-b322-110ae1590936\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"
Oct 30 12:46:59.126256 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.126223    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l982h\" (UniqueName: \"kubernetes.io/projected/fb963087-5281-48f9-b322-110ae1590936-kube-api-access-l982h\") pod \"payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv\" (UID: \"fb963087-5281-48f9-b322-110ae1590936\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"
Oct 30 12:46:59.142125 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.142100    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"
Oct 30 12:46:59.213794 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.213765    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"
Oct 30 12:46:59.473468 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:59.473435    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3929deb3_fa21_4e2f_8ada_6348c9faac3d.slice/crio-453ba294961401d4d86d75858810a38e69af10b75a6b73085110a0a7d1f12033 WatchSource:0}: Error finding container 453ba294961401d4d86d75858810a38e69af10b75a6b73085110a0a7d1f12033: Status 404 returned error can't find the container with id 453ba294961401d4d86d75858810a38e69af10b75a6b73085110a0a7d1f12033
Oct 30 12:46:59.478496 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.478465    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"]
Oct 30 12:46:59.513907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.513879    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"]
Oct 30 12:46:59.518973 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:46:59.518947    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb963087_5281_48f9_b322_110ae1590936.slice/crio-52bc769bee5e546d257d8bd79af8fb3bee0c777a7e3ebb16f881889b7d744289 WatchSource:0}: Error finding container 52bc769bee5e546d257d8bd79af8fb3bee0c777a7e3ebb16f881889b7d744289: Status 404 returned error can't find the container with id 52bc769bee5e546d257d8bd79af8fb3bee0c777a7e3ebb16f881889b7d744289
Oct 30 12:46:59.877588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.877547    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv" event={"ID":"3929deb3-fa21-4e2f-8ada-6348c9faac3d","Type":"ContainerStarted","Data":"79ce72ddf662e4084f6df23a705dba8e1653abdcb6b252feb3d5dcf026fb666f"}
Oct 30 12:46:59.878064 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.877610    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv" event={"ID":"3929deb3-fa21-4e2f-8ada-6348c9faac3d","Type":"ContainerStarted","Data":"453ba294961401d4d86d75858810a38e69af10b75a6b73085110a0a7d1f12033"}
Oct 30 12:46:59.878064 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.877653    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"
Oct 30 12:46:59.879264 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.879239    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv" event={"ID":"fb963087-5281-48f9-b322-110ae1590936","Type":"ContainerStarted","Data":"c1fe7e5a94a10a38b594db3ef4fb013beadda9db2b247ca4757e96cb6d774ed6"}
Oct 30 12:46:59.879421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.879284    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv" event={"ID":"fb963087-5281-48f9-b322-110ae1590936","Type":"ContainerStarted","Data":"52bc769bee5e546d257d8bd79af8fb3bee0c777a7e3ebb16f881889b7d744289"}
Oct 30 12:46:59.879421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.879343    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"
Oct 30 12:46:59.894074 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.894042    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"]
Oct 30 12:46:59.897813 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.897798    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"
Oct 30 12:46:59.913689 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.913669    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-4cd3fb-dockercfg-bqrvx\""
Oct 30 12:46:59.915747 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.915555    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"]
Oct 30 12:46:59.916712 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.916689    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-4cd3fb\""
Oct 30 12:46:59.926368 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.926326    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv" podStartSLOduration=1.926315932 podStartE2EDuration="1.926315932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:59.916759925 +0000 UTC m=+608.441044146" watchObservedRunningTime="2026-10-30 12:46:59.926315932 +0000 UTC m=+608.450600179"
Oct 30 12:46:59.957221 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.957175    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv" podStartSLOduration=1.957159876 podStartE2EDuration="1.957159876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:59.954788297 +0000 UTC m=+608.479072515" watchObservedRunningTime="2026-10-30 12:46:59.957159876 +0000 UTC m=+608.481444096"
Oct 30 12:46:59.976970 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.976943    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"]
Oct 30 12:46:59.981243 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.981221    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"
Oct 30 12:46:59.999730 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:46:59.999700    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"]
Oct 30 12:47:00.017568 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.017539    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwk4b\" (UniqueName: \"kubernetes.io/projected/e7f0fa26-8bb5-469d-9580-f460f4acc319-kube-api-access-vwk4b\") pod \"payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx\" (UID: \"e7f0fa26-8bb5-469d-9580-f460f4acc319\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"
Oct 30 12:47:00.017697 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.017574    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8be106-2a83-4465-9011-8298fff4f945-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b\" (UID: \"8e8be106-2a83-4465-9011-8298fff4f945\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"
Oct 30 12:47:00.017751 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.017686    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp48r\" (UniqueName: \"kubernetes.io/projected/8e8be106-2a83-4465-9011-8298fff4f945-kube-api-access-mp48r\") pod \"payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b\" (UID: \"8e8be106-2a83-4465-9011-8298fff4f945\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"
Oct 30 12:47:00.017751 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.017718    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f0fa26-8bb5-469d-9580-f460f4acc319-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx\" (UID: \"e7f0fa26-8bb5-469d-9580-f460f4acc319\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"
Oct 30 12:47:00.118752 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.118711    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwk4b\" (UniqueName: \"kubernetes.io/projected/e7f0fa26-8bb5-469d-9580-f460f4acc319-kube-api-access-vwk4b\") pod \"payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx\" (UID: \"e7f0fa26-8bb5-469d-9580-f460f4acc319\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"
Oct 30 12:47:00.118950 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.118765    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8be106-2a83-4465-9011-8298fff4f945-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b\" (UID: \"8e8be106-2a83-4465-9011-8298fff4f945\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"
Oct 30 12:47:00.118950 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.118818    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp48r\" (UniqueName: \"kubernetes.io/projected/8e8be106-2a83-4465-9011-8298fff4f945-kube-api-access-mp48r\") pod \"payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b\" (UID: \"8e8be106-2a83-4465-9011-8298fff4f945\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"
Oct 30 12:47:00.118950 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.118845    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f0fa26-8bb5-469d-9580-f460f4acc319-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx\" (UID: \"e7f0fa26-8bb5-469d-9580-f460f4acc319\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"
Oct 30 12:47:00.119748 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.119724    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f0fa26-8bb5-469d-9580-f460f4acc319-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx\" (UID: \"e7f0fa26-8bb5-469d-9580-f460f4acc319\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"
Oct 30 12:47:00.119845 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.119724    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8be106-2a83-4465-9011-8298fff4f945-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b\" (UID: \"8e8be106-2a83-4465-9011-8298fff4f945\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"
Oct 30 12:47:00.138266 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.138184    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp48r\" (UniqueName: \"kubernetes.io/projected/8e8be106-2a83-4465-9011-8298fff4f945-kube-api-access-mp48r\") pod \"payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b\" (UID: \"8e8be106-2a83-4465-9011-8298fff4f945\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"
Oct 30 12:47:00.138266 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.138255    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwk4b\" (UniqueName: \"kubernetes.io/projected/e7f0fa26-8bb5-469d-9580-f460f4acc319-kube-api-access-vwk4b\") pod \"payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx\" (UID: \"e7f0fa26-8bb5-469d-9580-f460f4acc319\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"
Oct 30 12:47:00.211255 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.211222    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"
Oct 30 12:47:00.293995 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.293962    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"
Oct 30 12:47:00.505112 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.505083    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"]
Oct 30 12:47:00.506938 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:47:00.506910    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8be106_2a83_4465_9011_8298fff4f945.slice/crio-7179f7d09de49ecc4f63fb24eed1d6597d64e3ab6140b33a89ad77930ad06025 WatchSource:0}: Error finding container 7179f7d09de49ecc4f63fb24eed1d6597d64e3ab6140b33a89ad77930ad06025: Status 404 returned error can't find the container with id 7179f7d09de49ecc4f63fb24eed1d6597d64e3ab6140b33a89ad77930ad06025
Oct 30 12:47:00.592376 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.592259    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"]
Oct 30 12:47:00.595002 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:47:00.594976    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f0fa26_8bb5_469d_9580_f460f4acc319.slice/crio-2b5ad6565ed9ef99e939b423616b629d803df9237a32c158a695681fb4662cfb WatchSource:0}: Error finding container 2b5ad6565ed9ef99e939b423616b629d803df9237a32c158a695681fb4662cfb: Status 404 returned error can't find the container with id 2b5ad6565ed9ef99e939b423616b629d803df9237a32c158a695681fb4662cfb
Oct 30 12:47:00.886864 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.886761    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b" event={"ID":"8e8be106-2a83-4465-9011-8298fff4f945","Type":"ContainerStarted","Data":"0240843b84f53d68c417ab7f1b01245319b0ddea3baca9fbdb3aa116ada10acf"}
Oct 30 12:47:00.886864 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.886815    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b" event={"ID":"8e8be106-2a83-4465-9011-8298fff4f945","Type":"ContainerStarted","Data":"7179f7d09de49ecc4f63fb24eed1d6597d64e3ab6140b33a89ad77930ad06025"}
Oct 30 12:47:00.887350 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.886959    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"
Oct 30 12:47:00.888453 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.888419    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx" event={"ID":"e7f0fa26-8bb5-469d-9580-f460f4acc319","Type":"ContainerStarted","Data":"084d36cb2e22ab5ef14859489ee505880e19f628f246096ba4f211804d281ae6"}
Oct 30 12:47:00.888605 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.888468    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx" event={"ID":"e7f0fa26-8bb5-469d-9580-f460f4acc319","Type":"ContainerStarted","Data":"2b5ad6565ed9ef99e939b423616b629d803df9237a32c158a695681fb4662cfb"}
Oct 30 12:47:00.888605 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.888522    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"
Oct 30 12:47:00.927916 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.927851    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b" podStartSLOduration=1.9278320899999999 podStartE2EDuration="1.92783209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:00.926350546 +0000 UTC m=+609.450634766" watchObservedRunningTime="2026-10-30 12:47:00.92783209 +0000 UTC m=+609.452116495"
Oct 30 12:47:00.965991 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:00.965944    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx" podStartSLOduration=1.965926346 podStartE2EDuration="1.965926346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:00.96354433 +0000 UTC m=+609.487828532" watchObservedRunningTime="2026-10-30 12:47:00.965926346 +0000 UTC m=+609.490210567"
Oct 30 12:47:03.827696 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:03.827664    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"
Oct 30 12:47:03.828221 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:03.827839    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"
Oct 30 12:47:08.874089 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:08.874058    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"
Oct 30 12:47:09.882705 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:09.882678    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"
Oct 30 12:47:10.126078 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.126041    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"]
Oct 30 12:47:10.130090 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.130067    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.151223 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.151195    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 12:47:10.151572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.151544    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 12:47:10.151666 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.151578    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 12:47:10.151666 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.151583    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jvqff\""
Oct 30 12:47:10.153863 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.153841    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"]
Oct 30 12:47:10.230852 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.230818    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.231041 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.230865    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.231041 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.230982    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.231041 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.231031    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.231162 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.231097    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2rr4\" (UniqueName: \"kubernetes.io/projected/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-kube-api-access-v2rr4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.231211 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.231149    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.231271 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.231252    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.332631 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.332598    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.332809 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.332686    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.332809 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.332715    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.332809 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.332764    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.332986 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.332818    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.332986 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.332844    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2rr4\" (UniqueName: \"kubernetes.io/projected/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-kube-api-access-v2rr4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.332986 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.332894    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.333596 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.333563    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.335483 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.335455    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.335599 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.335501    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.335907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.335886    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.335969 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.335888    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.353951 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.353918    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.354176 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.354152    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2rr4\" (UniqueName: \"kubernetes.io/projected/c5613511-ebe4-44bb-9d1b-a0652e3f97b8-kube-api-access-v2rr4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kvdnb\" (UID: \"c5613511-ebe4-44bb-9d1b-a0652e3f97b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.442835 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.442733    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:10.760450 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.760425    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"]
Oct 30 12:47:10.762358 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:47:10.762331    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5613511_ebe4_44bb_9d1b_a0652e3f97b8.slice/crio-806bdad834a0af67f122c670e1767645066ed6c74f16032d91d2bb4887423e2d WatchSource:0}: Error finding container 806bdad834a0af67f122c670e1767645066ed6c74f16032d91d2bb4887423e2d: Status 404 returned error can't find the container with id 806bdad834a0af67f122c670e1767645066ed6c74f16032d91d2bb4887423e2d
Oct 30 12:47:10.891293 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.891265    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"
Oct 30 12:47:10.891709 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.891338    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"
Oct 30 12:47:10.951882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:10.951853    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb" event={"ID":"c5613511-ebe4-44bb-9d1b-a0652e3f97b8","Type":"ContainerStarted","Data":"806bdad834a0af67f122c670e1767645066ed6c74f16032d91d2bb4887423e2d"}
Oct 30 12:47:11.898155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:11.897319    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"
Oct 30 12:47:11.898155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:11.897423    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"
Oct 30 12:47:13.219033 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:13.219003    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:47:13.219294 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:13.219071    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:47:14.021896 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:14.021754    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb" event={"ID":"c5613511-ebe4-44bb-9d1b-a0652e3f97b8","Type":"ContainerStarted","Data":"dac08181a0753c7b620af4b6c77a57d09221587955cb9b9186f9428b531339fe"}
Oct 30 12:47:14.023267 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:14.022925    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:14.024970 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:14.024936    5837 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kvdnb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:47:14.025089 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:14.025032    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb" podUID="c5613511-ebe4-44bb-9d1b-a0652e3f97b8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:47:14.091188 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:14.091101    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb" podStartSLOduration=1.637435736 podStartE2EDuration="4.091078259s" totalImagesPullingTime="2.453642523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:14.076963696 +0000 UTC m=+622.601247937" watchObservedRunningTime="2026-10-30 12:47:14.091078259 +0000 UTC m=+622.615362476"
Oct 30 12:47:15.027935 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:15.027904    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kvdnb"
Oct 30 12:47:15.124845 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:47:15.124811    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:47:23.074930 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:23.074897    5837 generic.go:356] "Generic (PLEG): container finished" podID="25644850-37a6-4820-8ed2-4f00fedd8274" containerID="23974a71b9810c8bd9b28fb891a606cff350e664ba3fda6e41453f8ed2a22c8a" exitCode=0
Oct 30 12:47:23.075387 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:23.074965    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" event={"ID":"25644850-37a6-4820-8ed2-4f00fedd8274","Type":"ContainerDied","Data":"23974a71b9810c8bd9b28fb891a606cff350e664ba3fda6e41453f8ed2a22c8a"}
Oct 30 12:47:23.075495 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:23.075436    5837 scope.go:122] "RemoveContainer" containerID="23974a71b9810c8bd9b28fb891a606cff350e664ba3fda6e41453f8ed2a22c8a"
Oct 30 12:47:23.076739 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:23.076709    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:47:23.076843 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:23.076824    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:47:23.076897 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:23.076880    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:47:24.082314 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:24.082277    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" event={"ID":"25644850-37a6-4820-8ed2-4f00fedd8274","Type":"ContainerStarted","Data":"1356c126cff4b48144c5c1a1fcd668ab8c162254826981b5b3cbb2adaf014e58"}
Oct 30 12:47:24.680093 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:24.680062    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:47:24.681040 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:24.681013    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:47:24.681153 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:24.681062    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" podUID="25644850-37a6-4820-8ed2-4f00fedd8274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 12:47:24.832834 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:24.832787    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:47:25.680270 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:25.680237    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:47:25.680762 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:25.680289    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" podUID="25644850-37a6-4820-8ed2-4f00fedd8274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 12:47:26.680288 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:26.680245    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:47:26.680719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:26.680319    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" podUID="25644850-37a6-4820-8ed2-4f00fedd8274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 12:47:27.680536 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:27.680504    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:47:27.680924 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:27.680558    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" podUID="25644850-37a6-4820-8ed2-4f00fedd8274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 12:47:28.680275 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:28.680245    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:47:28.680462 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:28.680291    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" podUID="25644850-37a6-4820-8ed2-4f00fedd8274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 12:47:29.684115 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:29.684080    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:47:29.685064 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:29.685043    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 12:47:45.094306 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:47:45.094269    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:47:56.319416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.318636    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"]
Oct 30 12:47:56.326417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.326133    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.347415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.347176    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-105228-openshift-default-dockercfg-kvxgn\""
Oct 30 12:47:56.388452 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.361886    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"]
Oct 30 12:47:56.517768 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.517167    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.517768 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.517215    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.517768 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.517245    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.517768 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.517318    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.517768 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.517362    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.517768 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.517446    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.517768 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.517468    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.517768 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.517514    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.517768 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.517534    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpmlk\" (UniqueName: \"kubernetes.io/projected/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-kube-api-access-bpmlk\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.618190 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.618098    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.618190 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.618136    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.618190 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.618179    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.618495 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.618203    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpmlk\" (UniqueName: \"kubernetes.io/projected/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-kube-api-access-bpmlk\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.618495 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.618272    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.618495 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.618299    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.618495 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.618327    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.618495 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.618463    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.618757 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.618514    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.618757 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.618580    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.618757 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.618686    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.618942 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.618912    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.619194 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.619151    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.619341 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.619315    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.621291 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.621268    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.621431 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.621345    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.637723 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.637697    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.637892 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.637870    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpmlk\" (UniqueName: \"kubernetes.io/projected/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-kube-api-access-bpmlk\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.653601 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.653574    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:56.997440 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:56.996901    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"]
Oct 30 12:47:57.016537 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:57.016492    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:47:57.016672 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:57.016600    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:47:57.016672 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:57.016643    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:47:57.309929 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:57.309806    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" event={"ID":"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd","Type":"ContainerStarted","Data":"cf30093d1663265d14fa7211b09ca5855854e6943f6bd85a78ac05905f891125"}
Oct 30 12:47:57.309929 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:57.309881    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" event={"ID":"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd","Type":"ContainerStarted","Data":"03a51cc1a46cf41d3769dfa07c3fd9928d533b29bd7dbff9db840d48f1894dcd"}
Oct 30 12:47:57.353554 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:57.352367    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" podStartSLOduration=1.352347785 podStartE2EDuration="1.352347785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:57.350446367 +0000 UTC m=+665.874730605" watchObservedRunningTime="2026-10-30 12:47:57.352347785 +0000 UTC m=+665.876632004"
Oct 30 12:47:57.654541 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:57.654452    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:47:58.656021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:58.655987    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 12:47:58.656428 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:58.656054    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" podUID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 12:47:59.654665 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:59.654629    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 12:47:59.654829 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:47:59.654694    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" podUID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 12:48:00.654129 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:00.654095    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 12:48:00.654549 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:00.654156    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" podUID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 12:48:01.654149 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:01.654116    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 12:48:01.654546 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:01.654188    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" podUID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 12:48:02.654539 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:02.654504    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 12:48:02.654927 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:02.654557    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" podUID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 12:48:03.751648 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:03.751613    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:48:03.752870 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:03.752844    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:48:03.753926 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:03.753905    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 12:48:07.220840 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.220808    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"]
Oct 30 12:48:07.224944 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.224924    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.240978 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.240959    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-4cd3fb-openshift-default-dockercfg-sd289\""
Oct 30 12:48:07.254513 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.254489    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"]
Oct 30 12:48:07.336402 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.336356    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.336582 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.336510    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982856f5-aa83-4e34-a199-ea238513db62-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.336582 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.336567    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982856f5-aa83-4e34-a199-ea238513db62-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.336896 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.336592    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982856f5-aa83-4e34-a199-ea238513db62-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.336896 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.336664    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.336983 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.336891    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.336983 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.336925    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2h4k\" (UniqueName: \"kubernetes.io/projected/982856f5-aa83-4e34-a199-ea238513db62-kube-api-access-g2h4k\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.337073 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.336984    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.337125 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.337107    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.437810 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.437785    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.437961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.437814    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2h4k\" (UniqueName: \"kubernetes.io/projected/982856f5-aa83-4e34-a199-ea238513db62-kube-api-access-g2h4k\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.437961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.437837    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.437961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.437862    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.437961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.437908    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.437961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.437943    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982856f5-aa83-4e34-a199-ea238513db62-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.438222 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.437993    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982856f5-aa83-4e34-a199-ea238513db62-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.438222 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.438024    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982856f5-aa83-4e34-a199-ea238513db62-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.438222 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.438056    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.438377 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.438259    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.438501 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.438472    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.438628 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.438517    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.438628 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.438557    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.438748 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.438733    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982856f5-aa83-4e34-a199-ea238513db62-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.440573 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.440554    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.440632 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.440570    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982856f5-aa83-4e34-a199-ea238513db62-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.456627 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.456597    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982856f5-aa83-4e34-a199-ea238513db62-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.457433 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.457411    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2h4k\" (UniqueName: \"kubernetes.io/projected/982856f5-aa83-4e34-a199-ea238513db62-kube-api-access-g2h4k\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.539660 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.539629    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:07.846293 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.846268    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"]
Oct 30 12:48:07.847656 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:48:07.847615    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982856f5_aa83_4e34_a199_ea238513db62.slice/crio-f53081db23b153261f511ab4f6c0b86ed80b9342a5cb72789a2a787267d6e741 WatchSource:0}: Error finding container f53081db23b153261f511ab4f6c0b86ed80b9342a5cb72789a2a787267d6e741: Status 404 returned error can't find the container with id f53081db23b153261f511ab4f6c0b86ed80b9342a5cb72789a2a787267d6e741
Oct 30 12:48:07.849968 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.849935    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:07.850046 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.850000    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:07.850046 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:07.850032    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:08.375037 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:08.374994    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" event={"ID":"982856f5-aa83-4e34-a199-ea238513db62","Type":"ContainerStarted","Data":"89b4990e50ad129fbacdfe09be4bd082fe7b3f551b4c00b90e3732b19e5fe1e1"}
Oct 30 12:48:08.375578 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:08.375442    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" event={"ID":"982856f5-aa83-4e34-a199-ea238513db62","Type":"ContainerStarted","Data":"f53081db23b153261f511ab4f6c0b86ed80b9342a5cb72789a2a787267d6e741"}
Oct 30 12:48:08.444998 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:08.444943    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" podStartSLOduration=1.4449271320000001 podStartE2EDuration="1.444927132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:08.443773161 +0000 UTC m=+676.968057379" watchObservedRunningTime="2026-10-30 12:48:08.444927132 +0000 UTC m=+676.969211351"
Oct 30 12:48:08.540580 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:08.540547    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:08.542056 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:08.542018    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:48:08.542216 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:08.542063    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 12:48:09.540122 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:09.540079    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:48:09.540665 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:09.540139    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 12:48:10.540990 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.540953    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:48:10.541354 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.541024    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 12:48:10.801446 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.801325    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"]
Oct 30 12:48:10.815446 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.815411    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.829028 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.828996    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"]
Oct 30 12:48:10.838445 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.838412    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c81f42-openshift-default-dockercfg-7fkzs\""
Oct 30 12:48:10.880334 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.880298    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.880547 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.880360    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.880547 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.880481    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.880646 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.880554    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.880646 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.880579    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.880646 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.880602    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.880753 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.880720    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncw6t\" (UniqueName: \"kubernetes.io/projected/bb740ce8-a013-43fd-b48c-96561ca1d3c5-kube-api-access-ncw6t\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.880788 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.880774    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.880858 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.880842    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.981794 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.981760    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncw6t\" (UniqueName: \"kubernetes.io/projected/bb740ce8-a013-43fd-b48c-96561ca1d3c5-kube-api-access-ncw6t\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.981971 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.981816    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.982047 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.982019    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.982192 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.982177    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.982239 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.982225    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.982270 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.982252    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.982303 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.982279    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.982357 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.982296    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.982357 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.982327    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.982510 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.982453    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.982831 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.982808    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.982933 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.982867    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.983049 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.983033    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:10.983119 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:10.983095    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:11.010004 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:11.009935    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:11.010952 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:11.010916    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:11.011568 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:11.011546    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncw6t\" (UniqueName: \"kubernetes.io/projected/bb740ce8-a013-43fd-b48c-96561ca1d3c5-kube-api-access-ncw6t\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:11.012490 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:11.012469    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:11.130038 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:11.129944    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:11.458385 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:11.458358    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"]
Oct 30 12:48:11.460242 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:48:11.460215    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb740ce8_a013_43fd_b48c_96561ca1d3c5.slice/crio-38ece60583adb7eab2aa28a0e1126546c78fe0a53ac81e708db70290c123160a WatchSource:0}: Error finding container 38ece60583adb7eab2aa28a0e1126546c78fe0a53ac81e708db70290c123160a: Status 404 returned error can't find the container with id 38ece60583adb7eab2aa28a0e1126546c78fe0a53ac81e708db70290c123160a
Oct 30 12:48:11.462654 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:11.462619    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:11.462758 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:11.462705    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:11.462758 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:11.462752    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:11.539939 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:11.539914    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:48:11.540053 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:11.539963    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 12:48:12.437720 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:12.437675    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w" event={"ID":"bb740ce8-a013-43fd-b48c-96561ca1d3c5","Type":"ContainerStarted","Data":"1de89d1ceee28b8e7a110d747a147e27255c4b2f663c467cd741470dcf262460"}
Oct 30 12:48:12.438179 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:12.437776    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w" event={"ID":"bb740ce8-a013-43fd-b48c-96561ca1d3c5","Type":"ContainerStarted","Data":"38ece60583adb7eab2aa28a0e1126546c78fe0a53ac81e708db70290c123160a"}
Oct 30 12:48:12.505442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:12.492656    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w" podStartSLOduration=2.492637673 podStartE2EDuration="2.492637673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:12.491954985 +0000 UTC m=+681.016239210" watchObservedRunningTime="2026-10-30 12:48:12.492637673 +0000 UTC m=+681.016921895"
Oct 30 12:48:12.540259 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:12.540195    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:48:12.540476 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:12.540267    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 12:48:13.133426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:13.133129    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:13.133426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:13.133260    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:48:13.133426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:13.133316    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w" podUID="bb740ce8-a013-43fd-b48c-96561ca1d3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 12:48:13.540664 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:13.540625    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:48:13.541170 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:13.540701    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 12:48:14.131051 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:14.131020    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:48:14.131216 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:14.131085    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w" podUID="bb740ce8-a013-43fd-b48c-96561ca1d3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 12:48:14.540926 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:14.540879    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:48:14.541315 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:14.540939    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 12:48:15.131754 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:15.131281    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:48:15.131754 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:15.131354    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w" podUID="bb740ce8-a013-43fd-b48c-96561ca1d3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 12:48:15.150425 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:48:15.148605    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:48:16.030422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.028043    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"]
Oct 30 12:48:16.060582 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.058780    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"]
Oct 30 12:48:16.060582 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.058907    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.110420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.094008    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-53dcb1-openshift-default-dockercfg-nlgks\""
Oct 30 12:48:16.150616 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.150066    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:48:16.150616 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.150130    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w" podUID="bb740ce8-a013-43fd-b48c-96561ca1d3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 12:48:16.250847 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.250797    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.251150 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.251126    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.251334 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.251316    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.251703 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.251679    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.251933 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.251914    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.252066 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.252050    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.252241 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.252225    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.252365 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.252350    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq5nw\" (UniqueName: \"kubernetes.io/projected/42ddf5f4-5772-45d0-8fbb-f1eba1931608-kube-api-access-jq5nw\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.252612 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.252592    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.253246 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.253226    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:16.254796 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.254779    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:16.256051 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.256034    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 12:48:16.355470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.354604    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.355470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.354704    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.355470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.354761    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.355470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.354856    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.355470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.354948    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.355470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.354996    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.355470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.355032    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.355470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.355150    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.355470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.355212    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5nw\" (UniqueName: \"kubernetes.io/projected/42ddf5f4-5772-45d0-8fbb-f1eba1931608-kube-api-access-jq5nw\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.356752 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.356729    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.356937 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.356854    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.357408 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.356624    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.357636 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.357573    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.357972 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.357944    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.359221 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.359166    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.361682 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.361659    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.379779 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.379745    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.380037 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.380010    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq5nw\" (UniqueName: \"kubernetes.io/projected/42ddf5f4-5772-45d0-8fbb-f1eba1931608-kube-api-access-jq5nw\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.463325 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.463288    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:16.941876 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.941603    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"]
Oct 30 12:48:16.944452 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:48:16.944420    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ddf5f4_5772_45d0_8fbb_f1eba1931608.slice/crio-5fdf4df2eb6b5b4d3ff14c8d7b1f34d091610056d29bf5239122295d21f33923 WatchSource:0}: Error finding container 5fdf4df2eb6b5b4d3ff14c8d7b1f34d091610056d29bf5239122295d21f33923: Status 404 returned error can't find the container with id 5fdf4df2eb6b5b4d3ff14c8d7b1f34d091610056d29bf5239122295d21f33923
Oct 30 12:48:16.947169 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.947127    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:16.947266 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.947224    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:16.947305 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:16.947270    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:17.130906 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:17.130869    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:48:17.131562 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:17.131526    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w" podUID="bb740ce8-a013-43fd-b48c-96561ca1d3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 12:48:17.652420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:17.652092    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" event={"ID":"42ddf5f4-5772-45d0-8fbb-f1eba1931608","Type":"ContainerStarted","Data":"853f2eab319a928f674fe993076a587a10a28af81f6c6221330b06e98f5d7748"}
Oct 30 12:48:17.652420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:17.652164    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" event={"ID":"42ddf5f4-5772-45d0-8fbb-f1eba1931608","Type":"ContainerStarted","Data":"5fdf4df2eb6b5b4d3ff14c8d7b1f34d091610056d29bf5239122295d21f33923"}
Oct 30 12:48:17.745420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:17.741815    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" podStartSLOduration=1.741789933 podStartE2EDuration="1.741789933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:17.724888566 +0000 UTC m=+686.249172787" watchObservedRunningTime="2026-10-30 12:48:17.741789933 +0000 UTC m=+686.266074154"
Oct 30 12:48:18.131057 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:18.131019    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:48:18.131491 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:18.131072    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w" podUID="bb740ce8-a013-43fd-b48c-96561ca1d3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 12:48:18.465420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:18.465296    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:18.466937 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:18.466908    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:48:18.467085 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:18.466956    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" podUID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 12:48:19.464768 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:19.464732    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:48:19.465175 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:19.464789    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" podUID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 12:48:19.713090 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:19.713046    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:19.713707 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:19.713646    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:19.714932 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:19.714874    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 12:48:20.465315 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:20.465274    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:48:20.465822 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:20.465332    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" podUID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 12:48:21.464842 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:21.464805    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:48:21.465016 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:21.464864    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" podUID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 12:48:22.464941 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:22.464902    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:48:22.465447 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:22.464966    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" podUID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 12:48:23.465362 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:23.465320    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:48:23.465812 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:23.465440    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" podUID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 12:48:25.054539 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:25.054511    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:25.055021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:25.054888    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:25.055555 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:25.055533    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 12:48:35.978943 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:35.978908    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"]
Oct 30 12:48:35.982821 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:35.982801    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"
Oct 30 12:48:36.001469 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.001442    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"]
Oct 30 12:48:36.010344 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.010314    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:48:36.010344 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.010336    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:48:36.010539 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.010379    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:48:36.010539 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.010460    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:48:36.057663 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.057627    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99188a77-6f30-45a7-b018-04f6288901e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e861da0-4gkmk\" (UID: \"99188a77-6f30-45a7-b018-04f6288901e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"
Oct 30 12:48:36.057828 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.057752    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99188a77-6f30-45a7-b018-04f6288901e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e861da0-4gkmk\" (UID: \"99188a77-6f30-45a7-b018-04f6288901e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"
Oct 30 12:48:36.057828 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.057777    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnf8c\" (UniqueName: \"kubernetes.io/projected/99188a77-6f30-45a7-b018-04f6288901e1-kube-api-access-gnf8c\") pod \"maas-api-revoke-sub-9e861da0-4gkmk\" (UID: \"99188a77-6f30-45a7-b018-04f6288901e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"
Oct 30 12:48:36.158557 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.158523    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99188a77-6f30-45a7-b018-04f6288901e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e861da0-4gkmk\" (UID: \"99188a77-6f30-45a7-b018-04f6288901e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"
Oct 30 12:48:36.158557 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.158561    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnf8c\" (UniqueName: \"kubernetes.io/projected/99188a77-6f30-45a7-b018-04f6288901e1-kube-api-access-gnf8c\") pod \"maas-api-revoke-sub-9e861da0-4gkmk\" (UID: \"99188a77-6f30-45a7-b018-04f6288901e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"
Oct 30 12:48:36.158792 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.158688    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99188a77-6f30-45a7-b018-04f6288901e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e861da0-4gkmk\" (UID: \"99188a77-6f30-45a7-b018-04f6288901e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"
Oct 30 12:48:36.159323 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.159295    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99188a77-6f30-45a7-b018-04f6288901e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e861da0-4gkmk\" (UID: \"99188a77-6f30-45a7-b018-04f6288901e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"
Oct 30 12:48:36.161086 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.161066    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99188a77-6f30-45a7-b018-04f6288901e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e861da0-4gkmk\" (UID: \"99188a77-6f30-45a7-b018-04f6288901e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"
Oct 30 12:48:36.178705 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.178677    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnf8c\" (UniqueName: \"kubernetes.io/projected/99188a77-6f30-45a7-b018-04f6288901e1-kube-api-access-gnf8c\") pod \"maas-api-revoke-sub-9e861da0-4gkmk\" (UID: \"99188a77-6f30-45a7-b018-04f6288901e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"
Oct 30 12:48:36.297296 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.297259    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"
Oct 30 12:48:36.589962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.589939    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"]
Oct 30 12:48:36.592009 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:48:36.591984    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99188a77_6f30_45a7_b018_04f6288901e1.slice/crio-fe7cbe9c6765a49e8bfb1d1cde0827aa0b495d1cee19b994874e27668568fcfc WatchSource:0}: Error finding container fe7cbe9c6765a49e8bfb1d1cde0827aa0b495d1cee19b994874e27668568fcfc: Status 404 returned error can't find the container with id fe7cbe9c6765a49e8bfb1d1cde0827aa0b495d1cee19b994874e27668568fcfc
Oct 30 12:48:36.808324 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:36.808289    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk" event={"ID":"99188a77-6f30-45a7-b018-04f6288901e1","Type":"ContainerStarted","Data":"fe7cbe9c6765a49e8bfb1d1cde0827aa0b495d1cee19b994874e27668568fcfc"}
Oct 30 12:48:37.815988 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:37.815904    5837 generic.go:356] "Generic (PLEG): container finished" podID="99188a77-6f30-45a7-b018-04f6288901e1" containerID="d968412018163c179d9cd3ae8cfa9ef695a00f271ddc57cad649c81392aca343" exitCode=0
Oct 30 12:48:37.816362 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:37.815990    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk" event={"ID":"99188a77-6f30-45a7-b018-04f6288901e1","Type":"ContainerDied","Data":"d968412018163c179d9cd3ae8cfa9ef695a00f271ddc57cad649c81392aca343"}
Oct 30 12:48:38.964833 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:38.964806    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"
Oct 30 12:48:39.100454 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:39.100334    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99188a77-6f30-45a7-b018-04f6288901e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99188a77-6f30-45a7-b018-04f6288901e1-maas-api-xks-ca\") pod \"99188a77-6f30-45a7-b018-04f6288901e1\" (UID: \"99188a77-6f30-45a7-b018-04f6288901e1\") "
Oct 30 12:48:39.100454 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:39.100422    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99188a77-6f30-45a7-b018-04f6288901e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99188a77-6f30-45a7-b018-04f6288901e1-maas-api-openshift-ca\") pod \"99188a77-6f30-45a7-b018-04f6288901e1\" (UID: \"99188a77-6f30-45a7-b018-04f6288901e1\") "
Oct 30 12:48:39.100454 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:39.100444    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99188a77-6f30-45a7-b018-04f6288901e1-kube-api-access-gnf8c\" (UniqueName: \"kubernetes.io/projected/99188a77-6f30-45a7-b018-04f6288901e1-kube-api-access-gnf8c\") pod \"99188a77-6f30-45a7-b018-04f6288901e1\" (UID: \"99188a77-6f30-45a7-b018-04f6288901e1\") "
Oct 30 12:48:39.100878 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:39.100845    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99188a77-6f30-45a7-b018-04f6288901e1-maas-api-openshift-ca" pod "99188a77-6f30-45a7-b018-04f6288901e1" (UID: "99188a77-6f30-45a7-b018-04f6288901e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:48:39.102681 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:39.102655    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99188a77-6f30-45a7-b018-04f6288901e1-kube-api-access-gnf8c" pod "99188a77-6f30-45a7-b018-04f6288901e1" (UID: "99188a77-6f30-45a7-b018-04f6288901e1"). InnerVolumeSpecName "kube-api-access-gnf8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:48:39.102757 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:39.102699    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99188a77-6f30-45a7-b018-04f6288901e1-maas-api-xks-ca" pod "99188a77-6f30-45a7-b018-04f6288901e1" (UID: "99188a77-6f30-45a7-b018-04f6288901e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:48:39.201735 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:39.201707    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99188a77-6f30-45a7-b018-04f6288901e1-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:48:39.201735 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:39.201733    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99188a77-6f30-45a7-b018-04f6288901e1-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:48:39.201930 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:39.201745    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnf8c\" (UniqueName: \"kubernetes.io/projected/99188a77-6f30-45a7-b018-04f6288901e1-kube-api-access-gnf8c\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:48:39.829608 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:39.829571    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk" event={"ID":"99188a77-6f30-45a7-b018-04f6288901e1","Type":"ContainerDied","Data":"fe7cbe9c6765a49e8bfb1d1cde0827aa0b495d1cee19b994874e27668568fcfc"}
Oct 30 12:48:39.829608 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:39.829598    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"
Oct 30 12:48:39.829808 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:39.829633    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe7cbe9c6765a49e8bfb1d1cde0827aa0b495d1cee19b994874e27668568fcfc"
Oct 30 12:48:41.695144 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.695102    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"]
Oct 30 12:48:41.696009 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.695975    5837 cpu_manager.go:414] "containerMap: removing container" podUID="99188a77-6f30-45a7-b018-04f6288901e1" containerName="revoke-keys"
Oct 30 12:48:41.696009 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.696002    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99188a77-6f30-45a7-b018-04f6288901e1" containerName="revoke-keys"
Oct 30 12:48:41.696178 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.696171    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="99188a77-6f30-45a7-b018-04f6288901e1" containerName="revoke-keys"
Oct 30 12:48:41.700808 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.700575    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.776461 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.769982    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"]
Oct 30 12:48:41.776461 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.772326    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"]
Oct 30 12:48:41.777260 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.776950    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"
Oct 30 12:48:41.833790 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.833308    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:48:41.833790 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.833439    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:48:41.833790 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.833506    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:48:41.833790 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.833605    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:48:41.841692 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.839815    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"]
Oct 30 12:48:41.876382 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.876337    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.878691 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.878662    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.878903 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.878886    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.879047 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.879030    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974423f8-8df0-4991-bde0-d09656d03d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2f7789-trb8v\" (UID: \"974423f8-8df0-4991-bde0-d09656d03d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"
Oct 30 12:48:41.879175 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.879160    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.879342 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.879327    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.879485 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.879469    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.879628 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.879611    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974423f8-8df0-4991-bde0-d09656d03d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2f7789-trb8v\" (UID: \"974423f8-8df0-4991-bde0-d09656d03d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"
Oct 30 12:48:41.879749 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.879734    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvqp4\" (UniqueName: \"kubernetes.io/projected/974423f8-8df0-4991-bde0-d09656d03d89-kube-api-access-hvqp4\") pod \"maas-api-revoke-sub-aa2f7789-trb8v\" (UID: \"974423f8-8df0-4991-bde0-d09656d03d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"
Oct 30 12:48:41.880183 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.879852    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84kg\" (UniqueName: \"kubernetes.io/projected/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-kube-api-access-l84kg\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.880183 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.879931    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.880183 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.879963    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.981146 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.981104    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.981337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.981157    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.981337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.981224    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.981337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.981270    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.981524 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.981371    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.981524 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.981464    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974423f8-8df0-4991-bde0-d09656d03d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2f7789-trb8v\" (UID: \"974423f8-8df0-4991-bde0-d09656d03d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"
Oct 30 12:48:41.981524 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.981503    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.981669 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.981593    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.981669 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.981631    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.981669 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.981660    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974423f8-8df0-4991-bde0-d09656d03d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2f7789-trb8v\" (UID: \"974423f8-8df0-4991-bde0-d09656d03d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"
Oct 30 12:48:41.981824 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.981693    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvqp4\" (UniqueName: \"kubernetes.io/projected/974423f8-8df0-4991-bde0-d09656d03d89-kube-api-access-hvqp4\") pod \"maas-api-revoke-sub-aa2f7789-trb8v\" (UID: \"974423f8-8df0-4991-bde0-d09656d03d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"
Oct 30 12:48:41.981824 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.981722    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84kg\" (UniqueName: \"kubernetes.io/projected/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-kube-api-access-l84kg\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.981976 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.981955    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.982052 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.981984    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.982634 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.982564    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974423f8-8df0-4991-bde0-d09656d03d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2f7789-trb8v\" (UID: \"974423f8-8df0-4991-bde0-d09656d03d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"
Oct 30 12:48:41.982784 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.982758    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.983128 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.983075    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:41.983928 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.983443    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:42.000452 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:41.997019    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:42.016114 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:42.016084    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974423f8-8df0-4991-bde0-d09656d03d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2f7789-trb8v\" (UID: \"974423f8-8df0-4991-bde0-d09656d03d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"
Oct 30 12:48:42.024511 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:42.023235    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84kg\" (UniqueName: \"kubernetes.io/projected/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-kube-api-access-l84kg\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:42.039415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:42.038956    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:42.054354 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:42.047423    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvqp4\" (UniqueName: \"kubernetes.io/projected/974423f8-8df0-4991-bde0-d09656d03d89-kube-api-access-hvqp4\") pod \"maas-api-revoke-sub-aa2f7789-trb8v\" (UID: \"974423f8-8df0-4991-bde0-d09656d03d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"
Oct 30 12:48:42.056451 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:42.055743    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:42.174482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:42.174447    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"
Oct 30 12:48:42.355975 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:42.331916    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:42.527482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:42.526508    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"]
Oct 30 12:48:42.732224 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:42.732173    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"]
Oct 30 12:48:42.738072 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:48:42.738016    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb107ba3d_a470_4c0b_a3bf_5b57ecafe0d9.slice/crio-a1bfa8cb0027e0d73f093d4b3228939429cef23c01dd3af949ad2f3774c58e4e WatchSource:0}: Error finding container a1bfa8cb0027e0d73f093d4b3228939429cef23c01dd3af949ad2f3774c58e4e: Status 404 returned error can't find the container with id a1bfa8cb0027e0d73f093d4b3228939429cef23c01dd3af949ad2f3774c58e4e
Oct 30 12:48:42.741485 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:42.741446    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:42.741593 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:42.741537    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:42.741593 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:42.741577    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:42.906422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:42.901807    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v" event={"ID":"974423f8-8df0-4991-bde0-d09656d03d89","Type":"ContainerStarted","Data":"4b5aa171b074292e99e380ce48a7c41835b34aa093cbf4c77c44c6fb8bcc4c55"}
Oct 30 12:48:42.914462 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:42.914377    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" event={"ID":"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9","Type":"ContainerStarted","Data":"0cbe8dddddf852091525970918b0aa97613358c180e32f3ac228a7c3bc0e3428"}
Oct 30 12:48:42.914657 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:42.914471    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" event={"ID":"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9","Type":"ContainerStarted","Data":"a1bfa8cb0027e0d73f093d4b3228939429cef23c01dd3af949ad2f3774c58e4e"}
Oct 30 12:48:43.017044 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:43.016984    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" podStartSLOduration=2.01696354 podStartE2EDuration="2.01696354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:43.009990726 +0000 UTC m=+711.534274958" watchObservedRunningTime="2026-10-30 12:48:43.01696354 +0000 UTC m=+711.541247760"
Oct 30 12:48:43.332984 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:43.332952    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:43.334527 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:43.334496    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:48:43.334686 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:43.334554    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 12:48:43.926252 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:43.926156    5837 generic.go:356] "Generic (PLEG): container finished" podID="974423f8-8df0-4991-bde0-d09656d03d89" containerID="94fcba83a55da8a9f3f38b017191e847049f184e29e20c1235df86a853371401" exitCode=0
Oct 30 12:48:43.926252 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:43.926237    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v" event={"ID":"974423f8-8df0-4991-bde0-d09656d03d89","Type":"ContainerDied","Data":"94fcba83a55da8a9f3f38b017191e847049f184e29e20c1235df86a853371401"}
Oct 30 12:48:44.332953 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:44.332919    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:48:44.333149 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:44.332983    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 12:48:45.117153 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:48:45.117119    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:48:45.123026 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:45.123007    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"
Oct 30 12:48:45.245118 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:45.245091    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/974423f8-8df0-4991-bde0-d09656d03d89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974423f8-8df0-4991-bde0-d09656d03d89-maas-api-xks-ca\") pod \"974423f8-8df0-4991-bde0-d09656d03d89\" (UID: \"974423f8-8df0-4991-bde0-d09656d03d89\") "
Oct 30 12:48:45.245318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:45.245135    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/974423f8-8df0-4991-bde0-d09656d03d89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974423f8-8df0-4991-bde0-d09656d03d89-maas-api-openshift-ca\") pod \"974423f8-8df0-4991-bde0-d09656d03d89\" (UID: \"974423f8-8df0-4991-bde0-d09656d03d89\") "
Oct 30 12:48:45.245318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:45.245230    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/974423f8-8df0-4991-bde0-d09656d03d89-kube-api-access-hvqp4\" (UniqueName: \"kubernetes.io/projected/974423f8-8df0-4991-bde0-d09656d03d89-kube-api-access-hvqp4\") pod \"974423f8-8df0-4991-bde0-d09656d03d89\" (UID: \"974423f8-8df0-4991-bde0-d09656d03d89\") "
Oct 30 12:48:45.245629 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:45.245599    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974423f8-8df0-4991-bde0-d09656d03d89-maas-api-openshift-ca" pod "974423f8-8df0-4991-bde0-d09656d03d89" (UID: "974423f8-8df0-4991-bde0-d09656d03d89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:48:45.247455 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:45.247365    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974423f8-8df0-4991-bde0-d09656d03d89-maas-api-xks-ca" pod "974423f8-8df0-4991-bde0-d09656d03d89" (UID: "974423f8-8df0-4991-bde0-d09656d03d89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:48:45.247455 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:45.247439    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974423f8-8df0-4991-bde0-d09656d03d89-kube-api-access-hvqp4" pod "974423f8-8df0-4991-bde0-d09656d03d89" (UID: "974423f8-8df0-4991-bde0-d09656d03d89"). InnerVolumeSpecName "kube-api-access-hvqp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:48:45.332989 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:45.332952    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:48:45.333165 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:45.333022    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 12:48:45.346648 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:45.346624    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvqp4\" (UniqueName: \"kubernetes.io/projected/974423f8-8df0-4991-bde0-d09656d03d89-kube-api-access-hvqp4\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:48:45.346648 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:45.346648    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974423f8-8df0-4991-bde0-d09656d03d89-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:48:45.346764 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:45.346658    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974423f8-8df0-4991-bde0-d09656d03d89-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:48:45.939624 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:45.939546    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"
Oct 30 12:48:45.939624 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:45.939564    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v" event={"ID":"974423f8-8df0-4991-bde0-d09656d03d89","Type":"ContainerDied","Data":"4b5aa171b074292e99e380ce48a7c41835b34aa093cbf4c77c44c6fb8bcc4c55"}
Oct 30 12:48:45.939624 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:45.939605    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b5aa171b074292e99e380ce48a7c41835b34aa093cbf4c77c44c6fb8bcc4c55"
Oct 30 12:48:46.149852 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.149819    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"]
Oct 30 12:48:46.150348 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.150335    5837 cpu_manager.go:414] "containerMap: removing container" podUID="974423f8-8df0-4991-bde0-d09656d03d89" containerName="revoke-keys"
Oct 30 12:48:46.150423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.150349    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="974423f8-8df0-4991-bde0-d09656d03d89" containerName="revoke-keys"
Oct 30 12:48:46.150463 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.150456    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="974423f8-8df0-4991-bde0-d09656d03d89" containerName="revoke-keys"
Oct 30 12:48:46.153909 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.153890    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.179832 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.179794    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"]
Oct 30 12:48:46.255370 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.255335    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.255546 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.255386    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.255546 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.255494    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351759f4-8556-4412-952a-a9a63f93c23a-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.255635 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.255569    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351759f4-8556-4412-952a-a9a63f93c23a-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.255635 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.255611    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsxjl\" (UniqueName: \"kubernetes.io/projected/351759f4-8556-4412-952a-a9a63f93c23a-kube-api-access-hsxjl\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.255704 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.255652    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.255704 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.255699    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.255761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.255738    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351759f4-8556-4412-952a-a9a63f93c23a-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.255851 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.255835    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.332847 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.332815    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:48:46.333009 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.332896    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 12:48:46.357529 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.357500    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351759f4-8556-4412-952a-a9a63f93c23a-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.357649 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.357539    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsxjl\" (UniqueName: \"kubernetes.io/projected/351759f4-8556-4412-952a-a9a63f93c23a-kube-api-access-hsxjl\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.357649 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.357597    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.357761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.357657    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.357761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.357678    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351759f4-8556-4412-952a-a9a63f93c23a-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.357761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.357744    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.357896 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.357829    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.357896 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.357866    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.357989 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.357900    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351759f4-8556-4412-952a-a9a63f93c23a-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.358171 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.358143    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.358245 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.358186    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.358317 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.358290    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.358383 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.358362    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.358694 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.358678    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351759f4-8556-4412-952a-a9a63f93c23a-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.360266 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.360246    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.360497 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.360479    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351759f4-8556-4412-952a-a9a63f93c23a-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.375849 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.375825    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351759f4-8556-4412-952a-a9a63f93c23a-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.376041 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.376024    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsxjl\" (UniqueName: \"kubernetes.io/projected/351759f4-8556-4412-952a-a9a63f93c23a-kube-api-access-hsxjl\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.474358 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.474322    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:46.817136 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.817110    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"]
Oct 30 12:48:46.817688 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:48:46.817667    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351759f4_8556_4412_952a_a9a63f93c23a.slice/crio-09e4468a2b182a705b652561b83fb1a67f1b175eebbf070bf4b16d21c57d8fd2 WatchSource:0}: Error finding container 09e4468a2b182a705b652561b83fb1a67f1b175eebbf070bf4b16d21c57d8fd2: Status 404 returned error can't find the container with id 09e4468a2b182a705b652561b83fb1a67f1b175eebbf070bf4b16d21c57d8fd2
Oct 30 12:48:46.820129 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.820095    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:46.820216 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.820157    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:46.820216 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.820186    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:46.948955 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.948926    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" event={"ID":"351759f4-8556-4412-952a-a9a63f93c23a","Type":"ContainerStarted","Data":"fa8ada98547aa343331b6af4d5932fb22fa08956d67800de17a7ba46412886a8"}
Oct 30 12:48:46.949143 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:46.948972    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" event={"ID":"351759f4-8556-4412-952a-a9a63f93c23a","Type":"ContainerStarted","Data":"09e4468a2b182a705b652561b83fb1a67f1b175eebbf070bf4b16d21c57d8fd2"}
Oct 30 12:48:47.333214 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:47.333182    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:48:47.333862 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:47.333285    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 12:48:47.475649 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:47.475596    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:47.476738 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:47.476478    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 12:48:47.476738 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:47.476525    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 12:48:48.333419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:48.332907    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:48:48.333419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:48.332970    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 12:48:48.475643 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:48.475602    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 12:48:48.475839 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:48.475678    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 12:48:49.332738 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:49.332704    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:48:49.332933 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:49.332755    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 12:48:49.474795 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:49.474758    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 12:48:49.475312 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:49.474808    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 12:48:50.337353 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:50.337323    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:50.337654 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:50.337631    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:50.338298 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:50.338282    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 12:48:50.410212 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:50.410154    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" podStartSLOduration=4.4101281629999995 podStartE2EDuration="4.410128163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:46.995728997 +0000 UTC m=+715.520013225" watchObservedRunningTime="2026-10-30 12:48:50.410128163 +0000 UTC m=+718.934412382"
Oct 30 12:48:50.475552 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:50.475514    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 12:48:50.475971 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:50.475585    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 12:48:51.475079 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:51.475042    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 12:48:51.475257 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:51.475098    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 12:48:52.370124 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.370090    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"]
Oct 30 12:48:52.375175 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.375150    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.422422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.415082    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"]
Oct 30 12:48:52.422422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.420132    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.422422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.420847    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"]
Oct 30 12:48:52.464527 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.464503    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"]
Oct 30 12:48:52.468953 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.468907    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.469100 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.468956    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.469100 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469075    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.469188 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469128    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.469188 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469170    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.469255 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469239    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48deaa90-e66d-454f-af97-1a5462424e82-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.469289 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469264    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.469322 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469308    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cdt2\" (UniqueName: \"kubernetes.io/projected/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-kube-api-access-5cdt2\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.469355 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469343    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k87lf\" (UniqueName: \"kubernetes.io/projected/48deaa90-e66d-454f-af97-1a5462424e82-kube-api-access-k87lf\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.469420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469382    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48deaa90-e66d-454f-af97-1a5462424e82-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.469459 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469445    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.469493 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469469    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.469526 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469489    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.469558 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469533    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48deaa90-e66d-454f-af97-1a5462424e82-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.469593 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469557    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.469633 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469618    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.469678 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469664    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.469751 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.469737    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.475297 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.475264    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 12:48:52.475455 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.475345    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 12:48:52.570983 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.570950    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.571144 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.570988    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.571144 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571020    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.571144 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571051    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.571144 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571069    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.571144 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571096    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48deaa90-e66d-454f-af97-1a5462424e82-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.571144 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571116    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.571337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571150    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cdt2\" (UniqueName: \"kubernetes.io/projected/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-kube-api-access-5cdt2\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.571337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571171    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k87lf\" (UniqueName: \"kubernetes.io/projected/48deaa90-e66d-454f-af97-1a5462424e82-kube-api-access-k87lf\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.571337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571204    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48deaa90-e66d-454f-af97-1a5462424e82-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.571337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571230    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.571337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571248    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.571337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571264    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.571337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571299    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48deaa90-e66d-454f-af97-1a5462424e82-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.571337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571318    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.571712 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571367    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.571712 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571423    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.571712 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571472    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.572035 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571946    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.572035 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.571958    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.572035 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.572035    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.572276 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.572071    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.572276 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.572097    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.573255 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.573229    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48deaa90-e66d-454f-af97-1a5462424e82-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.573750 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.573719    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.573949 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.573931    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.574095 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.574079    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.575432 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.574861    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.575432 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.574907    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.577981 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.577101    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48deaa90-e66d-454f-af97-1a5462424e82-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.602418 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.581731    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.609712 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.609679    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.611785 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.611755    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cdt2\" (UniqueName: \"kubernetes.io/projected/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-kube-api-access-5cdt2\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.613373 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.613346    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k87lf\" (UniqueName: \"kubernetes.io/projected/48deaa90-e66d-454f-af97-1a5462424e82-kube-api-access-k87lf\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.614808 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.614787    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48deaa90-e66d-454f-af97-1a5462424e82-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.615053 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.615032    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:52.706892 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.705632    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:52.745476 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:52.745452    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:53.097272 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:53.097235    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"]
Oct 30 12:48:53.097565 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:48:53.097534    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48deaa90_e66d_454f_af97_1a5462424e82.slice/crio-599510a4c8a88568ea61f5d601f8df8a06c2505ddcdee4d42720b21af871c108 WatchSource:0}: Error finding container 599510a4c8a88568ea61f5d601f8df8a06c2505ddcdee4d42720b21af871c108: Status 404 returned error can't find the container with id 599510a4c8a88568ea61f5d601f8df8a06c2505ddcdee4d42720b21af871c108
Oct 30 12:48:53.099954 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:53.099923    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:53.100033 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:53.099999    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:53.100033 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:53.100029    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:53.106520 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:53.106502    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"]
Oct 30 12:48:53.109326 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:48:53.109305    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7623e2_2633_4bb5_a6d9_f002c8ab8d01.slice/crio-eafff79a8ac2a725cc753facbb3af0bf85b83b2441fc52e399297d3836d24c78 WatchSource:0}: Error finding container eafff79a8ac2a725cc753facbb3af0bf85b83b2441fc52e399297d3836d24c78: Status 404 returned error can't find the container with id eafff79a8ac2a725cc753facbb3af0bf85b83b2441fc52e399297d3836d24c78
Oct 30 12:48:53.112121 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:53.112087    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:53.112219 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:53.112152    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:53.112219 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:53.112187    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:53.475236 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:53.475151    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 12:48:53.475236 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:53.475219    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 12:48:54.041427 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:54.039761    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" event={"ID":"48deaa90-e66d-454f-af97-1a5462424e82","Type":"ContainerStarted","Data":"64b7d7ab373443041363d3bb5d6152e8fc5c75532ba68ac8553786e55ba0ba42"}
Oct 30 12:48:54.041427 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:54.039828    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" event={"ID":"48deaa90-e66d-454f-af97-1a5462424e82","Type":"ContainerStarted","Data":"599510a4c8a88568ea61f5d601f8df8a06c2505ddcdee4d42720b21af871c108"}
Oct 30 12:48:54.044005 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:54.043977    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" event={"ID":"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01","Type":"ContainerStarted","Data":"c15f10603ebfeb5208d7085b75d9126d6eac6b670bb977f838fd42a48454499b"}
Oct 30 12:48:54.044160 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:54.044023    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" event={"ID":"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01","Type":"ContainerStarted","Data":"eafff79a8ac2a725cc753facbb3af0bf85b83b2441fc52e399297d3836d24c78"}
Oct 30 12:48:54.125420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:54.116298    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" podStartSLOduration=2.11627761 podStartE2EDuration="2.11627761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:54.095904931 +0000 UTC m=+722.620189150" watchObservedRunningTime="2026-10-30 12:48:54.11627761 +0000 UTC m=+722.640561830"
Oct 30 12:48:54.163496 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:54.163345    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" podStartSLOduration=2.163332312 podStartE2EDuration="2.163332312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:54.162629842 +0000 UTC m=+722.686914063" watchObservedRunningTime="2026-10-30 12:48:54.163332312 +0000 UTC m=+722.687616530"
Oct 30 12:48:54.483188 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:54.481502    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:54.483188 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:54.482735    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:54.503421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:54.495678    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 12:48:54.710203 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:54.710116    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:48:54.710838 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:54.710529    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 12:48:54.710838 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:54.710578    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" podUID="48deaa90-e66d-454f-af97-1a5462424e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 12:48:54.750643 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:54.750552    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:48:54.752429 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:54.752382    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:48:54.752598 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:54.752454    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 12:48:55.716852 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:55.716813    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 12:48:55.717491 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:55.717459    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" podUID="48deaa90-e66d-454f-af97-1a5462424e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 12:48:55.748783 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:55.748635    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:48:55.748783 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:55.748696    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 12:48:55.958422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:55.952808    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"]
Oct 30 12:48:55.963528 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:55.961356    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.006116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.006079    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.006471 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.006448    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.006707 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.006690    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a518b4-df4c-4b56-b56a-a979acf3ca15-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.006887 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.006870    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.007020 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.007006    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.031356 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.007182    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.031798 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.031770    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.031953 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.009225    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"]
Oct 30 12:48:56.032091 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.032045    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnhlk\" (UniqueName: \"kubernetes.io/projected/88a518b4-df4c-4b56-b56a-a979acf3ca15-kube-api-access-dnhlk\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.032323 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.032282    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.135161 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.135117    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.136023 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.135999    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.136700 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.136675    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnhlk\" (UniqueName: \"kubernetes.io/projected/88a518b4-df4c-4b56-b56a-a979acf3ca15-kube-api-access-dnhlk\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.137062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.137046    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.137283 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.137267    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.138087 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.137336    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.138087 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.135857    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.138329 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.138310    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.138526 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.138509    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a518b4-df4c-4b56-b56a-a979acf3ca15-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.138667 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.138630    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.138839 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.138821    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.138970 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.138953    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.139723 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.139162    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.139971 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.139204    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a518b4-df4c-4b56-b56a-a979acf3ca15-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.141475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.141458    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.141864 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.141845    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.155587 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.155519    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.155689 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.155631    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnhlk\" (UniqueName: \"kubernetes.io/projected/88a518b4-df4c-4b56-b56a-a979acf3ca15-kube-api-access-dnhlk\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.337385 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.337345    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:56.714484 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.710802    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 12:48:56.714484 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.710876    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" podUID="48deaa90-e66d-454f-af97-1a5462424e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 12:48:56.746965 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.746873    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:48:56.746965 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.746919    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 12:48:56.781431 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.781318    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"]
Oct 30 12:48:56.847284 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:48:56.836969    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a518b4_df4c_4b56_b56a_a979acf3ca15.slice/crio-8c05c9990876b3c7f5d4f475f4721e2da9a746236dd742b40e32b2edc63515f8 WatchSource:0}: Error finding container 8c05c9990876b3c7f5d4f475f4721e2da9a746236dd742b40e32b2edc63515f8: Status 404 returned error can't find the container with id 8c05c9990876b3c7f5d4f475f4721e2da9a746236dd742b40e32b2edc63515f8
Oct 30 12:48:56.847284 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.843343    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:56.847284 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.843473    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:56.847284 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:56.843532    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:48:57.078002 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:57.077969    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp" event={"ID":"88a518b4-df4c-4b56-b56a-a979acf3ca15","Type":"ContainerStarted","Data":"b9611a38f9ec36e924b2c3dc73dc6c4466b78b6da977651c581a6d627c86a616"}
Oct 30 12:48:57.078196 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:57.078020    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp" event={"ID":"88a518b4-df4c-4b56-b56a-a979acf3ca15","Type":"ContainerStarted","Data":"8c05c9990876b3c7f5d4f475f4721e2da9a746236dd742b40e32b2edc63515f8"}
Oct 30 12:48:57.131416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:57.130375    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp" podStartSLOduration=2.130357342 podStartE2EDuration="2.130357342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:57.129562195 +0000 UTC m=+725.653846432" watchObservedRunningTime="2026-10-30 12:48:57.130357342 +0000 UTC m=+725.654641563"
Oct 30 12:48:57.340442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:57.340288    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:48:57.340867 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:57.340745    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 12:48:57.340867 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:57.340805    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp" podUID="88a518b4-df4c-4b56-b56a-a979acf3ca15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 12:48:57.706854 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:57.706760    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 12:48:57.707510 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:57.707081    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" podUID="48deaa90-e66d-454f-af97-1a5462424e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 12:48:57.748039 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:57.748003    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:48:57.748690 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:57.748658    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 12:48:58.340419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:58.339834    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 12:48:58.340419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:58.339898    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp" podUID="88a518b4-df4c-4b56-b56a-a979acf3ca15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 12:48:58.710567 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:58.710477    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 12:48:58.710806 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:58.710780    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" podUID="48deaa90-e66d-454f-af97-1a5462424e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 12:48:58.746921 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:58.746798    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:48:58.746921 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:58.746861    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 12:48:59.341455 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:59.338756    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 12:48:59.341455 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:59.338818    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp" podUID="88a518b4-df4c-4b56-b56a-a979acf3ca15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 12:48:59.709480 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:59.708562    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 12:48:59.709480 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:59.708620    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" podUID="48deaa90-e66d-454f-af97-1a5462424e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 12:48:59.746524 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:59.746485    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:48:59.746705 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:48:59.746550    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 12:49:00.338248 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:00.338201    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 12:49:00.338498 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:00.338278    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp" podUID="88a518b4-df4c-4b56-b56a-a979acf3ca15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 12:49:00.716878 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:00.716776    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:49:00.717379 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:00.717191    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:49:00.718144 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:00.718119    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 12:49:00.751300 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:00.751270    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:49:00.751735 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:00.751716    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:49:00.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:00.752544    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 12:49:01.338034 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:01.337996    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 12:49:01.338243 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:01.338058    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp" podUID="88a518b4-df4c-4b56-b56a-a979acf3ca15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 12:49:02.337774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:02.337691    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 12:49:02.337774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:02.337756    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp" podUID="88a518b4-df4c-4b56-b56a-a979acf3ca15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 12:49:03.338007 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:03.337971    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 12:49:03.338607 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:03.338023    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp" podUID="88a518b4-df4c-4b56-b56a-a979acf3ca15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 12:49:04.342467 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:04.342439    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:49:04.342850 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:04.342837    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:49:04.343580 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:04.343560    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 12:49:07.025835 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.025808    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"]
Oct 30 12:49:07.029911 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.029894    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"
Oct 30 12:49:07.057816 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.057789    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:49:07.057995 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.057821    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:49:07.058099 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.057914    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:49:07.058171 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.057987    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:49:07.060819 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.060795    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"]
Oct 30 12:49:07.079588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.078465    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hwp6\" (UniqueName: \"kubernetes.io/projected/71746b8c-2788-4d44-bbce-10fea0de7d57-kube-api-access-9hwp6\") pod \"maas-api-revoke-sub-590a9ce4-pwh6z\" (UID: \"71746b8c-2788-4d44-bbce-10fea0de7d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"
Oct 30 12:49:07.079588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.078566    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71746b8c-2788-4d44-bbce-10fea0de7d57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590a9ce4-pwh6z\" (UID: \"71746b8c-2788-4d44-bbce-10fea0de7d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"
Oct 30 12:49:07.079588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.078820    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71746b8c-2788-4d44-bbce-10fea0de7d57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590a9ce4-pwh6z\" (UID: \"71746b8c-2788-4d44-bbce-10fea0de7d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"
Oct 30 12:49:07.180170 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.180139    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71746b8c-2788-4d44-bbce-10fea0de7d57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590a9ce4-pwh6z\" (UID: \"71746b8c-2788-4d44-bbce-10fea0de7d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"
Oct 30 12:49:07.180350 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.180273    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hwp6\" (UniqueName: \"kubernetes.io/projected/71746b8c-2788-4d44-bbce-10fea0de7d57-kube-api-access-9hwp6\") pod \"maas-api-revoke-sub-590a9ce4-pwh6z\" (UID: \"71746b8c-2788-4d44-bbce-10fea0de7d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"
Oct 30 12:49:07.180483 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.180349    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71746b8c-2788-4d44-bbce-10fea0de7d57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590a9ce4-pwh6z\" (UID: \"71746b8c-2788-4d44-bbce-10fea0de7d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"
Oct 30 12:49:07.181000 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.180928    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71746b8c-2788-4d44-bbce-10fea0de7d57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590a9ce4-pwh6z\" (UID: \"71746b8c-2788-4d44-bbce-10fea0de7d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"
Oct 30 12:49:07.182700 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.182678    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71746b8c-2788-4d44-bbce-10fea0de7d57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590a9ce4-pwh6z\" (UID: \"71746b8c-2788-4d44-bbce-10fea0de7d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"
Oct 30 12:49:07.199944 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.199917    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hwp6\" (UniqueName: \"kubernetes.io/projected/71746b8c-2788-4d44-bbce-10fea0de7d57-kube-api-access-9hwp6\") pod \"maas-api-revoke-sub-590a9ce4-pwh6z\" (UID: \"71746b8c-2788-4d44-bbce-10fea0de7d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"
Oct 30 12:49:07.330920 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.330835    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"]
Oct 30 12:49:07.335114 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.335092    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"
Oct 30 12:49:07.343574 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.343551    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"
Oct 30 12:49:07.353021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.352999    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"]
Oct 30 12:49:07.382257 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.382225    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wgrs\" (UniqueName: \"kubernetes.io/projected/34f61b3a-ae4c-4506-b9ff-d95bce390791-kube-api-access-6wgrs\") pod \"maas-api-revoke-sub-6dfac4a7-2rkb6\" (UID: \"34f61b3a-ae4c-4506-b9ff-d95bce390791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"
Oct 30 12:49:07.382496 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.382464    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f61b3a-ae4c-4506-b9ff-d95bce390791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dfac4a7-2rkb6\" (UID: \"34f61b3a-ae4c-4506-b9ff-d95bce390791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"
Oct 30 12:49:07.382496 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.382490    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f61b3a-ae4c-4506-b9ff-d95bce390791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dfac4a7-2rkb6\" (UID: \"34f61b3a-ae4c-4506-b9ff-d95bce390791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"
Oct 30 12:49:07.483920 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.483886    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f61b3a-ae4c-4506-b9ff-d95bce390791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dfac4a7-2rkb6\" (UID: \"34f61b3a-ae4c-4506-b9ff-d95bce390791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"
Oct 30 12:49:07.483920 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.483922    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f61b3a-ae4c-4506-b9ff-d95bce390791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dfac4a7-2rkb6\" (UID: \"34f61b3a-ae4c-4506-b9ff-d95bce390791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"
Oct 30 12:49:07.484165 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.483954    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wgrs\" (UniqueName: \"kubernetes.io/projected/34f61b3a-ae4c-4506-b9ff-d95bce390791-kube-api-access-6wgrs\") pod \"maas-api-revoke-sub-6dfac4a7-2rkb6\" (UID: \"34f61b3a-ae4c-4506-b9ff-d95bce390791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"
Oct 30 12:49:07.484580 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.484553    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f61b3a-ae4c-4506-b9ff-d95bce390791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dfac4a7-2rkb6\" (UID: \"34f61b3a-ae4c-4506-b9ff-d95bce390791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"
Oct 30 12:49:07.486443 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.486426    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f61b3a-ae4c-4506-b9ff-d95bce390791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dfac4a7-2rkb6\" (UID: \"34f61b3a-ae4c-4506-b9ff-d95bce390791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"
Oct 30 12:49:07.503282 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.503261    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wgrs\" (UniqueName: \"kubernetes.io/projected/34f61b3a-ae4c-4506-b9ff-d95bce390791-kube-api-access-6wgrs\") pod \"maas-api-revoke-sub-6dfac4a7-2rkb6\" (UID: \"34f61b3a-ae4c-4506-b9ff-d95bce390791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"
Oct 30 12:49:07.644729 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.644697    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"]
Oct 30 12:49:07.645500 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:49:07.645474    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71746b8c_2788_4d44_bbce_10fea0de7d57.slice/crio-97534195c8275034ef692d1fdcac556143a1f417c70a8aec93eb0cfd6ddabc02 WatchSource:0}: Error finding container 97534195c8275034ef692d1fdcac556143a1f417c70a8aec93eb0cfd6ddabc02: Status 404 returned error can't find the container with id 97534195c8275034ef692d1fdcac556143a1f417c70a8aec93eb0cfd6ddabc02
Oct 30 12:49:07.649213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.649195    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"
Oct 30 12:49:07.944427 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:07.944378    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"]
Oct 30 12:49:07.945200 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:49:07.945172    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f61b3a_ae4c_4506_b9ff_d95bce390791.slice/crio-39836b4b68db52d540946282b1eac653730f98b09200384a0ce1988fe4970e30 WatchSource:0}: Error finding container 39836b4b68db52d540946282b1eac653730f98b09200384a0ce1988fe4970e30: Status 404 returned error can't find the container with id 39836b4b68db52d540946282b1eac653730f98b09200384a0ce1988fe4970e30
Oct 30 12:49:08.176197 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:08.176087    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z" event={"ID":"71746b8c-2788-4d44-bbce-10fea0de7d57","Type":"ContainerStarted","Data":"353d0110cebecfebe75120b0fe58eca60de4750974ae82d3117307016f9b2058"}
Oct 30 12:49:08.176197 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:08.176150    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z" event={"ID":"71746b8c-2788-4d44-bbce-10fea0de7d57","Type":"ContainerStarted","Data":"97534195c8275034ef692d1fdcac556143a1f417c70a8aec93eb0cfd6ddabc02"}
Oct 30 12:49:08.177449 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:08.177424    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6" event={"ID":"34f61b3a-ae4c-4506-b9ff-d95bce390791","Type":"ContainerStarted","Data":"39836b4b68db52d540946282b1eac653730f98b09200384a0ce1988fe4970e30"}
Oct 30 12:49:08.214080 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:08.214029    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z" podStartSLOduration=1.214014601 podStartE2EDuration="1.214014601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:08.212696964 +0000 UTC m=+736.736981202" watchObservedRunningTime="2026-10-30 12:49:08.214014601 +0000 UTC m=+736.738298820"
Oct 30 12:49:09.185612 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:09.185515    5837 generic.go:356] "Generic (PLEG): container finished" podID="34f61b3a-ae4c-4506-b9ff-d95bce390791" containerID="39c6271bd09fe83604889ab39058891cd3b1849dfb606f4c782199bbb47e4f07" exitCode=0
Oct 30 12:49:09.185612 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:09.185598    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6" event={"ID":"34f61b3a-ae4c-4506-b9ff-d95bce390791","Type":"ContainerDied","Data":"39c6271bd09fe83604889ab39058891cd3b1849dfb606f4c782199bbb47e4f07"}
Oct 30 12:49:09.187239 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:09.187211    5837 generic.go:356] "Generic (PLEG): container finished" podID="71746b8c-2788-4d44-bbce-10fea0de7d57" containerID="353d0110cebecfebe75120b0fe58eca60de4750974ae82d3117307016f9b2058" exitCode=0
Oct 30 12:49:09.187367 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:09.187271    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z" event={"ID":"71746b8c-2788-4d44-bbce-10fea0de7d57","Type":"ContainerDied","Data":"353d0110cebecfebe75120b0fe58eca60de4750974ae82d3117307016f9b2058"}
Oct 30 12:49:10.359579 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.359547    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"
Oct 30 12:49:10.395874 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.395847    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"
Oct 30 12:49:10.429080 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.429046    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71746b8c-2788-4d44-bbce-10fea0de7d57-kube-api-access-9hwp6\" (UniqueName: \"kubernetes.io/projected/71746b8c-2788-4d44-bbce-10fea0de7d57-kube-api-access-9hwp6\") pod \"71746b8c-2788-4d44-bbce-10fea0de7d57\" (UID: \"71746b8c-2788-4d44-bbce-10fea0de7d57\") "
Oct 30 12:49:10.429260 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.429113    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71746b8c-2788-4d44-bbce-10fea0de7d57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71746b8c-2788-4d44-bbce-10fea0de7d57-maas-api-xks-ca\") pod \"71746b8c-2788-4d44-bbce-10fea0de7d57\" (UID: \"71746b8c-2788-4d44-bbce-10fea0de7d57\") "
Oct 30 12:49:10.429260 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.429153    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71746b8c-2788-4d44-bbce-10fea0de7d57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71746b8c-2788-4d44-bbce-10fea0de7d57-maas-api-openshift-ca\") pod \"71746b8c-2788-4d44-bbce-10fea0de7d57\" (UID: \"71746b8c-2788-4d44-bbce-10fea0de7d57\") "
Oct 30 12:49:10.429946 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.429861    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71746b8c-2788-4d44-bbce-10fea0de7d57-maas-api-openshift-ca" pod "71746b8c-2788-4d44-bbce-10fea0de7d57" (UID: "71746b8c-2788-4d44-bbce-10fea0de7d57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:49:10.431628 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.431605    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71746b8c-2788-4d44-bbce-10fea0de7d57-kube-api-access-9hwp6" pod "71746b8c-2788-4d44-bbce-10fea0de7d57" (UID: "71746b8c-2788-4d44-bbce-10fea0de7d57"). InnerVolumeSpecName "kube-api-access-9hwp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:49:10.431942 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.431924    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71746b8c-2788-4d44-bbce-10fea0de7d57-maas-api-xks-ca" pod "71746b8c-2788-4d44-bbce-10fea0de7d57" (UID: "71746b8c-2788-4d44-bbce-10fea0de7d57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:49:10.530123 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.530094    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34f61b3a-ae4c-4506-b9ff-d95bce390791-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f61b3a-ae4c-4506-b9ff-d95bce390791-maas-api-xks-ca\") pod \"34f61b3a-ae4c-4506-b9ff-d95bce390791\" (UID: \"34f61b3a-ae4c-4506-b9ff-d95bce390791\") "
Oct 30 12:49:10.530288 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.530197    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f61b3a-ae4c-4506-b9ff-d95bce390791-kube-api-access-6wgrs\" (UniqueName: \"kubernetes.io/projected/34f61b3a-ae4c-4506-b9ff-d95bce390791-kube-api-access-6wgrs\") pod \"34f61b3a-ae4c-4506-b9ff-d95bce390791\" (UID: \"34f61b3a-ae4c-4506-b9ff-d95bce390791\") "
Oct 30 12:49:10.530288 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.530219    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f61b3a-ae4c-4506-b9ff-d95bce390791-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f61b3a-ae4c-4506-b9ff-d95bce390791-maas-api-openshift-ca\") pod \"34f61b3a-ae4c-4506-b9ff-d95bce390791\" (UID: \"34f61b3a-ae4c-4506-b9ff-d95bce390791\") "
Oct 30 12:49:10.530660 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.530634    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f61b3a-ae4c-4506-b9ff-d95bce390791-maas-api-openshift-ca" pod "34f61b3a-ae4c-4506-b9ff-d95bce390791" (UID: "34f61b3a-ae4c-4506-b9ff-d95bce390791"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:49:10.530757 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.530665    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hwp6\" (UniqueName: \"kubernetes.io/projected/71746b8c-2788-4d44-bbce-10fea0de7d57-kube-api-access-9hwp6\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:10.530757 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.530683    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71746b8c-2788-4d44-bbce-10fea0de7d57-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:10.530757 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.530693    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71746b8c-2788-4d44-bbce-10fea0de7d57-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:10.532185 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.532164    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34f61b3a-ae4c-4506-b9ff-d95bce390791-maas-api-xks-ca" pod "34f61b3a-ae4c-4506-b9ff-d95bce390791" (UID: "34f61b3a-ae4c-4506-b9ff-d95bce390791"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:49:10.532352 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.532329    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f61b3a-ae4c-4506-b9ff-d95bce390791-kube-api-access-6wgrs" pod "34f61b3a-ae4c-4506-b9ff-d95bce390791" (UID: "34f61b3a-ae4c-4506-b9ff-d95bce390791"). InnerVolumeSpecName "kube-api-access-6wgrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:49:10.632206 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.632169    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f61b3a-ae4c-4506-b9ff-d95bce390791-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:10.632206 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.632199    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wgrs\" (UniqueName: \"kubernetes.io/projected/34f61b3a-ae4c-4506-b9ff-d95bce390791-kube-api-access-6wgrs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:10.632206 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.632211    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f61b3a-ae4c-4506-b9ff-d95bce390791-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:10.987338 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.987307    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"]
Oct 30 12:49:10.987989 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.987970    5837 cpu_manager.go:414] "containerMap: removing container" podUID="34f61b3a-ae4c-4506-b9ff-d95bce390791" containerName="revoke-keys"
Oct 30 12:49:10.988088 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.987991    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f61b3a-ae4c-4506-b9ff-d95bce390791" containerName="revoke-keys"
Oct 30 12:49:10.988088 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.988039    5837 cpu_manager.go:414] "containerMap: removing container" podUID="71746b8c-2788-4d44-bbce-10fea0de7d57" containerName="revoke-keys"
Oct 30 12:49:10.988088 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.988047    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71746b8c-2788-4d44-bbce-10fea0de7d57" containerName="revoke-keys"
Oct 30 12:49:10.988243 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.988213    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="71746b8c-2788-4d44-bbce-10fea0de7d57" containerName="revoke-keys"
Oct 30 12:49:10.988243 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.988230    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f61b3a-ae4c-4506-b9ff-d95bce390791" containerName="revoke-keys"
Oct 30 12:49:10.991927 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:10.991906    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.015960 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.015930    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"]
Oct 30 12:49:11.037000 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.036944    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.037226 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.037040    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.037226 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.037066    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.037226 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.037106    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.037226 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.037128    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.037226 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.037164    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.037226 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.037212    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7lj\" (UniqueName: \"kubernetes.io/projected/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-kube-api-access-7d7lj\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.037678 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.037287    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.037678 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.037410    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.139205 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.139169    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.139958 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.139931    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.140167 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.140148    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.140347 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.140319    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7lj\" (UniqueName: \"kubernetes.io/projected/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-kube-api-access-7d7lj\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.140503 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.140435    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.140574 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.140548    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.140639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.140608    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.140723 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.140700    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.140768 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.140728    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.140990 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.140960    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.141067 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.139872    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.142571 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.142538    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.142909 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.142886    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.143349 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.143321    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.144076 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.144052    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.146104 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.146080    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.172304 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.172279    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7lj\" (UniqueName: \"kubernetes.io/projected/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-kube-api-access-7d7lj\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.174750 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.174733    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.200725 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.200695    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"
Oct 30 12:49:11.200905 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.200726    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z" event={"ID":"71746b8c-2788-4d44-bbce-10fea0de7d57","Type":"ContainerDied","Data":"97534195c8275034ef692d1fdcac556143a1f417c70a8aec93eb0cfd6ddabc02"}
Oct 30 12:49:11.200905 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.200779    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97534195c8275034ef692d1fdcac556143a1f417c70a8aec93eb0cfd6ddabc02"
Oct 30 12:49:11.202188 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.202160    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6" event={"ID":"34f61b3a-ae4c-4506-b9ff-d95bce390791","Type":"ContainerDied","Data":"39836b4b68db52d540946282b1eac653730f98b09200384a0ce1988fe4970e30"}
Oct 30 12:49:11.202312 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.202206    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39836b4b68db52d540946282b1eac653730f98b09200384a0ce1988fe4970e30"
Oct 30 12:49:11.202312 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.202215    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"
Oct 30 12:49:11.305680 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.305585    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:11.387236 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.387205    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"]
Oct 30 12:49:11.393042 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.393019    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"
Oct 30 12:49:11.407867 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.407612    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:49:11.407867 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.407721    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:49:11.407867 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.407779    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:49:11.408102 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.407873    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:49:11.409426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.409377    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"]
Oct 30 12:49:11.445112 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.445083    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n9dj\" (UniqueName: \"kubernetes.io/projected/93d644d9-84a3-4f85-96f2-417a7e7ae36c-kube-api-access-2n9dj\") pod \"maas-api-revoke-sub-44c2ec46-fdxbn\" (UID: \"93d644d9-84a3-4f85-96f2-417a7e7ae36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"
Oct 30 12:49:11.445262 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.445201    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d644d9-84a3-4f85-96f2-417a7e7ae36c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c2ec46-fdxbn\" (UID: \"93d644d9-84a3-4f85-96f2-417a7e7ae36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"
Oct 30 12:49:11.445429 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.445384    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d644d9-84a3-4f85-96f2-417a7e7ae36c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c2ec46-fdxbn\" (UID: \"93d644d9-84a3-4f85-96f2-417a7e7ae36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"
Oct 30 12:49:11.546540 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.546507    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d644d9-84a3-4f85-96f2-417a7e7ae36c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c2ec46-fdxbn\" (UID: \"93d644d9-84a3-4f85-96f2-417a7e7ae36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"
Oct 30 12:49:11.546540 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.546543    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n9dj\" (UniqueName: \"kubernetes.io/projected/93d644d9-84a3-4f85-96f2-417a7e7ae36c-kube-api-access-2n9dj\") pod \"maas-api-revoke-sub-44c2ec46-fdxbn\" (UID: \"93d644d9-84a3-4f85-96f2-417a7e7ae36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"
Oct 30 12:49:11.546722 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.546593    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d644d9-84a3-4f85-96f2-417a7e7ae36c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c2ec46-fdxbn\" (UID: \"93d644d9-84a3-4f85-96f2-417a7e7ae36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"
Oct 30 12:49:11.547126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.547106    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d644d9-84a3-4f85-96f2-417a7e7ae36c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c2ec46-fdxbn\" (UID: \"93d644d9-84a3-4f85-96f2-417a7e7ae36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"
Oct 30 12:49:11.548996 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.548978    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d644d9-84a3-4f85-96f2-417a7e7ae36c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c2ec46-fdxbn\" (UID: \"93d644d9-84a3-4f85-96f2-417a7e7ae36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"
Oct 30 12:49:11.566363 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.566306    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n9dj\" (UniqueName: \"kubernetes.io/projected/93d644d9-84a3-4f85-96f2-417a7e7ae36c-kube-api-access-2n9dj\") pod \"maas-api-revoke-sub-44c2ec46-fdxbn\" (UID: \"93d644d9-84a3-4f85-96f2-417a7e7ae36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"
Oct 30 12:49:11.613845 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.613820    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"]
Oct 30 12:49:11.614740 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:49:11.614716    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6194a6_2f9c_4dc6_b517_8c819b850f3a.slice/crio-7a910af4a34b2c2580b0691d1aef904bd6df5e4bcaea8de780afed6803feedc5 WatchSource:0}: Error finding container 7a910af4a34b2c2580b0691d1aef904bd6df5e4bcaea8de780afed6803feedc5: Status 404 returned error can't find the container with id 7a910af4a34b2c2580b0691d1aef904bd6df5e4bcaea8de780afed6803feedc5
Oct 30 12:49:11.617093 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.617065    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:11.617194 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.617123    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:11.617194 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.617152    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:11.713531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:11.713498    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"
Oct 30 12:49:12.023422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:12.016702    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"]
Oct 30 12:49:12.219182 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:12.219153    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn" event={"ID":"93d644d9-84a3-4f85-96f2-417a7e7ae36c","Type":"ContainerStarted","Data":"3570ccc7d98be0a1b17b2ff5d6e78d1987ed9473b024505c5d2fc2140e5f65aa"}
Oct 30 12:49:12.220789 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:12.220763    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" event={"ID":"bf6194a6-2f9c-4dc6-b517-8c819b850f3a","Type":"ContainerStarted","Data":"2b672a2041c0b2733c3c8702d20c71f1d430f6f5a6b7a45dc87a080260b40172"}
Oct 30 12:49:12.220918 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:12.220813    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" event={"ID":"bf6194a6-2f9c-4dc6-b517-8c819b850f3a","Type":"ContainerStarted","Data":"7a910af4a34b2c2580b0691d1aef904bd6df5e4bcaea8de780afed6803feedc5"}
Oct 30 12:49:12.264077 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:12.263974    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podStartSLOduration=2.2639601 podStartE2EDuration="2.2639601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:12.261819251 +0000 UTC m=+740.786103471" watchObservedRunningTime="2026-10-30 12:49:12.2639601 +0000 UTC m=+740.788244320"
Oct 30 12:49:12.308613 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:12.308572    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:12.310384 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:12.310360    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 12:49:12.310489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:12.310429    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 12:49:13.229068 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:13.229033    5837 generic.go:356] "Generic (PLEG): container finished" podID="93d644d9-84a3-4f85-96f2-417a7e7ae36c" containerID="615e389c883408a88bc9a8bc6dbbe7b1c637facd5e80d5f81b445df6d0cde591" exitCode=0
Oct 30 12:49:13.229473 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:13.229073    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn" event={"ID":"93d644d9-84a3-4f85-96f2-417a7e7ae36c","Type":"ContainerDied","Data":"615e389c883408a88bc9a8bc6dbbe7b1c637facd5e80d5f81b445df6d0cde591"}
Oct 30 12:49:13.308820 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:13.308786    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 12:49:13.308964 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:13.308841    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 12:49:14.308873 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:14.308839    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 12:49:14.309330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:14.309302    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 12:49:14.437891 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:14.437867    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"
Oct 30 12:49:14.510032 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:14.509934    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d644d9-84a3-4f85-96f2-417a7e7ae36c-kube-api-access-2n9dj\" (UniqueName: \"kubernetes.io/projected/93d644d9-84a3-4f85-96f2-417a7e7ae36c-kube-api-access-2n9dj\") pod \"93d644d9-84a3-4f85-96f2-417a7e7ae36c\" (UID: \"93d644d9-84a3-4f85-96f2-417a7e7ae36c\") "
Oct 30 12:49:14.510355 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:14.510329    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d644d9-84a3-4f85-96f2-417a7e7ae36c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d644d9-84a3-4f85-96f2-417a7e7ae36c-maas-api-openshift-ca\") pod \"93d644d9-84a3-4f85-96f2-417a7e7ae36c\" (UID: \"93d644d9-84a3-4f85-96f2-417a7e7ae36c\") "
Oct 30 12:49:14.510599 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:14.510579    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93d644d9-84a3-4f85-96f2-417a7e7ae36c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d644d9-84a3-4f85-96f2-417a7e7ae36c-maas-api-xks-ca\") pod \"93d644d9-84a3-4f85-96f2-417a7e7ae36c\" (UID: \"93d644d9-84a3-4f85-96f2-417a7e7ae36c\") "
Oct 30 12:49:14.511810 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:14.511779    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d644d9-84a3-4f85-96f2-417a7e7ae36c-maas-api-openshift-ca" pod "93d644d9-84a3-4f85-96f2-417a7e7ae36c" (UID: "93d644d9-84a3-4f85-96f2-417a7e7ae36c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:49:14.515091 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:14.515061    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d644d9-84a3-4f85-96f2-417a7e7ae36c-kube-api-access-2n9dj" pod "93d644d9-84a3-4f85-96f2-417a7e7ae36c" (UID: "93d644d9-84a3-4f85-96f2-417a7e7ae36c"). InnerVolumeSpecName "kube-api-access-2n9dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:49:14.515558 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:14.515520    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93d644d9-84a3-4f85-96f2-417a7e7ae36c-maas-api-xks-ca" pod "93d644d9-84a3-4f85-96f2-417a7e7ae36c" (UID: "93d644d9-84a3-4f85-96f2-417a7e7ae36c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:49:14.612037 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:14.612010    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d644d9-84a3-4f85-96f2-417a7e7ae36c-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:14.612037 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:14.612035    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d644d9-84a3-4f85-96f2-417a7e7ae36c-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:14.612037 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:14.612046    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n9dj\" (UniqueName: \"kubernetes.io/projected/93d644d9-84a3-4f85-96f2-417a7e7ae36c-kube-api-access-2n9dj\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:15.118576 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:49:15.118541    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:49:15.243030 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:15.243000    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"
Oct 30 12:49:15.243179 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:15.243001    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn" event={"ID":"93d644d9-84a3-4f85-96f2-417a7e7ae36c","Type":"ContainerDied","Data":"3570ccc7d98be0a1b17b2ff5d6e78d1987ed9473b024505c5d2fc2140e5f65aa"}
Oct 30 12:49:15.243179 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:15.243123    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3570ccc7d98be0a1b17b2ff5d6e78d1987ed9473b024505c5d2fc2140e5f65aa"
Oct 30 12:49:15.308420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:15.307866    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 12:49:15.308420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:15.307930    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 12:49:16.308942 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:16.308892    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 12:49:16.309512 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:16.308962    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 12:49:17.311495 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:17.310846    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 12:49:17.311495 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:17.310917    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 12:49:18.306326 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:18.306289    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 12:49:18.306743 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:18.306714    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 12:49:20.339577 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:20.339529    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:49:20.341995 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:20.341960    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:21.411560 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:21.411520    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:49:21.412140 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:21.412109    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:21.592765 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:21.592729    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:21.593518 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:21.593484    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:21.594455 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:21.594434    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 12:49:25.275585 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.275553    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"]
Oct 30 12:49:25.276143 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.276130    5837 cpu_manager.go:414] "containerMap: removing container" podUID="93d644d9-84a3-4f85-96f2-417a7e7ae36c" containerName="revoke-keys"
Oct 30 12:49:25.276185 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.276144    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d644d9-84a3-4f85-96f2-417a7e7ae36c" containerName="revoke-keys"
Oct 30 12:49:25.276231 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.276222    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d644d9-84a3-4f85-96f2-417a7e7ae36c" containerName="revoke-keys"
Oct 30 12:49:25.281540 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.281519    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"
Oct 30 12:49:25.297554 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.297533    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"]
Oct 30 12:49:25.306460 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.306354    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:49:25.306460 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.306433    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:49:25.308682 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.308662    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:49:25.308682 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.308677    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:49:25.331522 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.331487    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt9px\" (UniqueName: \"kubernetes.io/projected/b4ff0f3d-11af-46b6-a565-187e8b7207e1-kube-api-access-rt9px\") pod \"maas-api-revoke-sub-8bc34e4d-cpzbg\" (UID: \"b4ff0f3d-11af-46b6-a565-187e8b7207e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"
Oct 30 12:49:25.331624 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.331539    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ff0f3d-11af-46b6-a565-187e8b7207e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bc34e4d-cpzbg\" (UID: \"b4ff0f3d-11af-46b6-a565-187e8b7207e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"
Oct 30 12:49:25.331733 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.331719    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ff0f3d-11af-46b6-a565-187e8b7207e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bc34e4d-cpzbg\" (UID: \"b4ff0f3d-11af-46b6-a565-187e8b7207e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"
Oct 30 12:49:25.432556 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.432529    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt9px\" (UniqueName: \"kubernetes.io/projected/b4ff0f3d-11af-46b6-a565-187e8b7207e1-kube-api-access-rt9px\") pod \"maas-api-revoke-sub-8bc34e4d-cpzbg\" (UID: \"b4ff0f3d-11af-46b6-a565-187e8b7207e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"
Oct 30 12:49:25.432672 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.432569    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ff0f3d-11af-46b6-a565-187e8b7207e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bc34e4d-cpzbg\" (UID: \"b4ff0f3d-11af-46b6-a565-187e8b7207e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"
Oct 30 12:49:25.432672 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.432666    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ff0f3d-11af-46b6-a565-187e8b7207e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bc34e4d-cpzbg\" (UID: \"b4ff0f3d-11af-46b6-a565-187e8b7207e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"
Oct 30 12:49:25.433175 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.433154    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ff0f3d-11af-46b6-a565-187e8b7207e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bc34e4d-cpzbg\" (UID: \"b4ff0f3d-11af-46b6-a565-187e8b7207e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"
Oct 30 12:49:25.435045 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.435025    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ff0f3d-11af-46b6-a565-187e8b7207e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bc34e4d-cpzbg\" (UID: \"b4ff0f3d-11af-46b6-a565-187e8b7207e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"
Oct 30 12:49:25.451771 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.451744    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt9px\" (UniqueName: \"kubernetes.io/projected/b4ff0f3d-11af-46b6-a565-187e8b7207e1-kube-api-access-rt9px\") pod \"maas-api-revoke-sub-8bc34e4d-cpzbg\" (UID: \"b4ff0f3d-11af-46b6-a565-187e8b7207e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"
Oct 30 12:49:25.595128 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.595056    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"
Oct 30 12:49:25.900647 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:25.900616    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"]
Oct 30 12:49:25.907643 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:49:25.907606    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ff0f3d_11af_46b6_a565_187e8b7207e1.slice/crio-d52d321f324c463fed73ca2a04e5509f0b84a33b92fd86d329db69b667fce691 WatchSource:0}: Error finding container d52d321f324c463fed73ca2a04e5509f0b84a33b92fd86d329db69b667fce691: Status 404 returned error can't find the container with id d52d321f324c463fed73ca2a04e5509f0b84a33b92fd86d329db69b667fce691
Oct 30 12:49:26.438281 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:26.438244    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg" event={"ID":"b4ff0f3d-11af-46b6-a565-187e8b7207e1","Type":"ContainerStarted","Data":"4af1f75df7231c0e554793fcef59086760c9edabc1be3d5f91348b2ec7389698"}
Oct 30 12:49:26.438790 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:26.438302    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg" event={"ID":"b4ff0f3d-11af-46b6-a565-187e8b7207e1","Type":"ContainerStarted","Data":"d52d321f324c463fed73ca2a04e5509f0b84a33b92fd86d329db69b667fce691"}
Oct 30 12:49:26.476177 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:26.476084    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg" podStartSLOduration=1.476069473 podStartE2EDuration="1.476069473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:26.474775884 +0000 UTC m=+754.999060103" watchObservedRunningTime="2026-10-30 12:49:26.476069473 +0000 UTC m=+755.000353692"
Oct 30 12:49:27.455424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:27.455365    5837 generic.go:356] "Generic (PLEG): container finished" podID="b4ff0f3d-11af-46b6-a565-187e8b7207e1" containerID="4af1f75df7231c0e554793fcef59086760c9edabc1be3d5f91348b2ec7389698" exitCode=0
Oct 30 12:49:27.455918 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:27.455489    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg" event={"ID":"b4ff0f3d-11af-46b6-a565-187e8b7207e1","Type":"ContainerDied","Data":"4af1f75df7231c0e554793fcef59086760c9edabc1be3d5f91348b2ec7389698"}
Oct 30 12:49:28.945716 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:28.945681    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"
Oct 30 12:49:29.023175 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:29.023120    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4ff0f3d-11af-46b6-a565-187e8b7207e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ff0f3d-11af-46b6-a565-187e8b7207e1-maas-api-xks-ca\") pod \"b4ff0f3d-11af-46b6-a565-187e8b7207e1\" (UID: \"b4ff0f3d-11af-46b6-a565-187e8b7207e1\") "
Oct 30 12:49:29.024440 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:29.024412    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ff0f3d-11af-46b6-a565-187e8b7207e1-kube-api-access-rt9px\" (UniqueName: \"kubernetes.io/projected/b4ff0f3d-11af-46b6-a565-187e8b7207e1-kube-api-access-rt9px\") pod \"b4ff0f3d-11af-46b6-a565-187e8b7207e1\" (UID: \"b4ff0f3d-11af-46b6-a565-187e8b7207e1\") "
Oct 30 12:49:29.024693 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:29.024675    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ff0f3d-11af-46b6-a565-187e8b7207e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ff0f3d-11af-46b6-a565-187e8b7207e1-maas-api-openshift-ca\") pod \"b4ff0f3d-11af-46b6-a565-187e8b7207e1\" (UID: \"b4ff0f3d-11af-46b6-a565-187e8b7207e1\") "
Oct 30 12:49:29.026507 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:29.026429    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ff0f3d-11af-46b6-a565-187e8b7207e1-maas-api-openshift-ca" pod "b4ff0f3d-11af-46b6-a565-187e8b7207e1" (UID: "b4ff0f3d-11af-46b6-a565-187e8b7207e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:49:29.053064 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:29.053009    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ff0f3d-11af-46b6-a565-187e8b7207e1-kube-api-access-rt9px" pod "b4ff0f3d-11af-46b6-a565-187e8b7207e1" (UID: "b4ff0f3d-11af-46b6-a565-187e8b7207e1"). InnerVolumeSpecName "kube-api-access-rt9px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:49:29.077427 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:29.077362    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ff0f3d-11af-46b6-a565-187e8b7207e1-maas-api-xks-ca" pod "b4ff0f3d-11af-46b6-a565-187e8b7207e1" (UID: "b4ff0f3d-11af-46b6-a565-187e8b7207e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:49:29.148637 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:29.129485    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt9px\" (UniqueName: \"kubernetes.io/projected/b4ff0f3d-11af-46b6-a565-187e8b7207e1-kube-api-access-rt9px\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:29.148637 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:29.129519    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ff0f3d-11af-46b6-a565-187e8b7207e1-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:29.148637 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:29.129535    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ff0f3d-11af-46b6-a565-187e8b7207e1-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:29.515431 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:29.501872    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg" event={"ID":"b4ff0f3d-11af-46b6-a565-187e8b7207e1","Type":"ContainerDied","Data":"d52d321f324c463fed73ca2a04e5509f0b84a33b92fd86d329db69b667fce691"}
Oct 30 12:49:29.515431 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:29.501935    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d52d321f324c463fed73ca2a04e5509f0b84a33b92fd86d329db69b667fce691"
Oct 30 12:49:29.515431 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:29.502004    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"
Oct 30 12:49:32.054794 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.054761    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"]
Oct 30 12:49:32.055752 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.055724    5837 cpu_manager.go:414] "containerMap: removing container" podUID="b4ff0f3d-11af-46b6-a565-187e8b7207e1" containerName="revoke-keys"
Oct 30 12:49:32.055752 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.055748    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ff0f3d-11af-46b6-a565-187e8b7207e1" containerName="revoke-keys"
Oct 30 12:49:32.055943 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.055898    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ff0f3d-11af-46b6-a565-187e8b7207e1" containerName="revoke-keys"
Oct 30 12:49:32.060458 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.060438    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"
Oct 30 12:49:32.076147 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.076123    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"]
Oct 30 12:49:32.084178 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.084149    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:49:32.084299 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.084186    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:49:32.089583 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.089559    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:49:32.089700 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.089603    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:49:32.136957 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.136927    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d0005f0-7kqml\" (UID: \"2d7ffb87-9c58-44f7-a83a-ef0d6866e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"
Oct 30 12:49:32.137128 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.136964    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z92rf\" (UniqueName: \"kubernetes.io/projected/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-kube-api-access-z92rf\") pod \"maas-api-revoke-sub-7d0005f0-7kqml\" (UID: \"2d7ffb87-9c58-44f7-a83a-ef0d6866e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"
Oct 30 12:49:32.137128 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.137094    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d0005f0-7kqml\" (UID: \"2d7ffb87-9c58-44f7-a83a-ef0d6866e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"
Oct 30 12:49:32.238506 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.238477    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d0005f0-7kqml\" (UID: \"2d7ffb87-9c58-44f7-a83a-ef0d6866e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"
Oct 30 12:49:32.238667 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.238510    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z92rf\" (UniqueName: \"kubernetes.io/projected/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-kube-api-access-z92rf\") pod \"maas-api-revoke-sub-7d0005f0-7kqml\" (UID: \"2d7ffb87-9c58-44f7-a83a-ef0d6866e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"
Oct 30 12:49:32.238667 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.238639    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d0005f0-7kqml\" (UID: \"2d7ffb87-9c58-44f7-a83a-ef0d6866e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"
Oct 30 12:49:32.239349 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.239318    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d0005f0-7kqml\" (UID: \"2d7ffb87-9c58-44f7-a83a-ef0d6866e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"
Oct 30 12:49:32.241608 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.241587    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d0005f0-7kqml\" (UID: \"2d7ffb87-9c58-44f7-a83a-ef0d6866e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"
Oct 30 12:49:32.258100 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.258075    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z92rf\" (UniqueName: \"kubernetes.io/projected/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-kube-api-access-z92rf\") pod \"maas-api-revoke-sub-7d0005f0-7kqml\" (UID: \"2d7ffb87-9c58-44f7-a83a-ef0d6866e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"
Oct 30 12:49:32.375298 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.375228    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"
Oct 30 12:49:32.678200 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:32.678166    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"]
Oct 30 12:49:33.206386 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.206353    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"]
Oct 30 12:49:33.210544 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.210527    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:33.225406 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.225377    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f9689ecfae68\""
Oct 30 12:49:33.225746 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.225729    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-68396bdfc185\""
Oct 30 12:49:33.225832 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.225744    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-dddj8\""
Oct 30 12:49:33.225832 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.225781    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 12:49:33.237024 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.236996    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 12:49:33.237305 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.237242    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 12:49:33.237305 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.237277    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 12:49:33.240442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.240419    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"]
Oct 30 12:49:33.253453 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.253382    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wtwt\" (UniqueName: \"kubernetes.io/projected/0c77ff27-6464-4cb8-875d-01a2dc4744fa-kube-api-access-7wtwt\") pod \"payload-processing-external-model-5c9d5b9b75-tpcg4\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") " pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:33.253553 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.253471    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c77ff27-6464-4cb8-875d-01a2dc4744fa-provider-credentials\") pod \"payload-processing-external-model-5c9d5b9b75-tpcg4\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") " pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:33.253553 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.253514    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c77ff27-6464-4cb8-875d-01a2dc4744fa-routing-overlay-volume\") pod \"payload-processing-external-model-5c9d5b9b75-tpcg4\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") " pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:33.253643 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.253601    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c77ff27-6464-4cb8-875d-01a2dc4744fa-plugins-config-volume\") pod \"payload-processing-external-model-5c9d5b9b75-tpcg4\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") " pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:33.356417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.355499    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c77ff27-6464-4cb8-875d-01a2dc4744fa-routing-overlay-volume\") pod \"payload-processing-external-model-5c9d5b9b75-tpcg4\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") " pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:33.356417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.355584    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c77ff27-6464-4cb8-875d-01a2dc4744fa-plugins-config-volume\") pod \"payload-processing-external-model-5c9d5b9b75-tpcg4\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") " pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:33.356417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.355854    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wtwt\" (UniqueName: \"kubernetes.io/projected/0c77ff27-6464-4cb8-875d-01a2dc4744fa-kube-api-access-7wtwt\") pod \"payload-processing-external-model-5c9d5b9b75-tpcg4\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") " pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:33.356417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.355890    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c77ff27-6464-4cb8-875d-01a2dc4744fa-provider-credentials\") pod \"payload-processing-external-model-5c9d5b9b75-tpcg4\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") " pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:33.357421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.357295    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c77ff27-6464-4cb8-875d-01a2dc4744fa-plugins-config-volume\") pod \"payload-processing-external-model-5c9d5b9b75-tpcg4\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") " pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:33.358445 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.358084    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c77ff27-6464-4cb8-875d-01a2dc4744fa-routing-overlay-volume\") pod \"payload-processing-external-model-5c9d5b9b75-tpcg4\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") " pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:33.377032 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.377002    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c77ff27-6464-4cb8-875d-01a2dc4744fa-provider-credentials\") pod \"payload-processing-external-model-5c9d5b9b75-tpcg4\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") " pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:33.382224 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.382154    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wtwt\" (UniqueName: \"kubernetes.io/projected/0c77ff27-6464-4cb8-875d-01a2dc4744fa-kube-api-access-7wtwt\") pod \"payload-processing-external-model-5c9d5b9b75-tpcg4\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") " pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:33.524711 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.524626    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:33.560475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.560377    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml" event={"ID":"2d7ffb87-9c58-44f7-a83a-ef0d6866e951","Type":"ContainerStarted","Data":"ffefadfca48e5f0f2c351e5cf035ac2d660e865802b3c2ab538c81a03e3a1a9b"}
Oct 30 12:49:33.560641 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.560515    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml" event={"ID":"2d7ffb87-9c58-44f7-a83a-ef0d6866e951","Type":"ContainerStarted","Data":"9b76b7eed40bf5025c01f728d4eaece74e9bd8110fa3384d48f422fb5d4a90df"}
Oct 30 12:49:33.600872 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.600691    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml" podStartSLOduration=1.600671968 podStartE2EDuration="1.600671968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:33.599949633 +0000 UTC m=+762.124233854" watchObservedRunningTime="2026-10-30 12:49:33.600671968 +0000 UTC m=+762.124956188"
Oct 30 12:49:33.840449 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:33.840417    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"]
Oct 30 12:49:33.841694 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:49:33.841668    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c77ff27_6464_4cb8_875d_01a2dc4744fa.slice/crio-84a84010445af6f22f2f7ac11c896676e1e3663f47952baff0576f833a32ad73 WatchSource:0}: Error finding container 84a84010445af6f22f2f7ac11c896676e1e3663f47952baff0576f833a32ad73: Status 404 returned error can't find the container with id 84a84010445af6f22f2f7ac11c896676e1e3663f47952baff0576f833a32ad73
Oct 30 12:49:34.570333 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:34.570230    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4" event={"ID":"0c77ff27-6464-4cb8-875d-01a2dc4744fa","Type":"ContainerStarted","Data":"f5261520e97e1cb38492d48f6f842906eb821aea472dfc75d95157ebbb2f8f1b"}
Oct 30 12:49:34.570333 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:34.570305    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4" event={"ID":"0c77ff27-6464-4cb8-875d-01a2dc4744fa","Type":"ContainerStarted","Data":"84a84010445af6f22f2f7ac11c896676e1e3663f47952baff0576f833a32ad73"}
Oct 30 12:49:34.570824 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:34.570340    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:34.572329 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:34.572303    5837 generic.go:356] "Generic (PLEG): container finished" podID="2d7ffb87-9c58-44f7-a83a-ef0d6866e951" containerID="ffefadfca48e5f0f2c351e5cf035ac2d660e865802b3c2ab538c81a03e3a1a9b" exitCode=0
Oct 30 12:49:34.572481 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:34.572345    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml" event={"ID":"2d7ffb87-9c58-44f7-a83a-ef0d6866e951","Type":"ContainerDied","Data":"ffefadfca48e5f0f2c351e5cf035ac2d660e865802b3c2ab538c81a03e3a1a9b"}
Oct 30 12:49:34.629419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:34.624451    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4" podStartSLOduration=1.624351852 podStartE2EDuration="1.624351852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:34.622773437 +0000 UTC m=+763.147057659" watchObservedRunningTime="2026-10-30 12:49:34.624351852 +0000 UTC m=+763.148636076"
Oct 30 12:49:35.668466 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:35.668435    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"]
Oct 30 12:49:35.677732 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:35.677561    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"]
Oct 30 12:49:35.677732 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:35.677661    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"
Oct 30 12:49:35.757327 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:35.757289    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sk5s\" (UniqueName: \"kubernetes.io/projected/6af17d7f-a451-48d7-a03d-c25693a988f7-kube-api-access-7sk5s\") pod \"maas-api-revoke-sub-e3caae70-bwjqt\" (UID: \"6af17d7f-a451-48d7-a03d-c25693a988f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"
Oct 30 12:49:35.763866 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:35.763833    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af17d7f-a451-48d7-a03d-c25693a988f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3caae70-bwjqt\" (UID: \"6af17d7f-a451-48d7-a03d-c25693a988f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"
Oct 30 12:49:35.764085 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:35.764066    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af17d7f-a451-48d7-a03d-c25693a988f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3caae70-bwjqt\" (UID: \"6af17d7f-a451-48d7-a03d-c25693a988f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"
Oct 30 12:49:35.868912 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:35.867244    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sk5s\" (UniqueName: \"kubernetes.io/projected/6af17d7f-a451-48d7-a03d-c25693a988f7-kube-api-access-7sk5s\") pod \"maas-api-revoke-sub-e3caae70-bwjqt\" (UID: \"6af17d7f-a451-48d7-a03d-c25693a988f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"
Oct 30 12:49:35.868912 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:35.867457    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af17d7f-a451-48d7-a03d-c25693a988f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3caae70-bwjqt\" (UID: \"6af17d7f-a451-48d7-a03d-c25693a988f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"
Oct 30 12:49:35.868912 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:35.867498    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af17d7f-a451-48d7-a03d-c25693a988f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3caae70-bwjqt\" (UID: \"6af17d7f-a451-48d7-a03d-c25693a988f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"
Oct 30 12:49:35.894516 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:35.872608    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af17d7f-a451-48d7-a03d-c25693a988f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3caae70-bwjqt\" (UID: \"6af17d7f-a451-48d7-a03d-c25693a988f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"
Oct 30 12:49:35.980084 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:35.980058    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sk5s\" (UniqueName: \"kubernetes.io/projected/6af17d7f-a451-48d7-a03d-c25693a988f7-kube-api-access-7sk5s\") pod \"maas-api-revoke-sub-e3caae70-bwjqt\" (UID: \"6af17d7f-a451-48d7-a03d-c25693a988f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"
Oct 30 12:49:35.982804 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:35.982782    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af17d7f-a451-48d7-a03d-c25693a988f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3caae70-bwjqt\" (UID: \"6af17d7f-a451-48d7-a03d-c25693a988f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"
Oct 30 12:49:36.061431 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.046186    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"
Oct 30 12:49:36.069290 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.069264    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"
Oct 30 12:49:36.181900 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.181807    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-maas-api-xks-ca\") pod \"2d7ffb87-9c58-44f7-a83a-ef0d6866e951\" (UID: \"2d7ffb87-9c58-44f7-a83a-ef0d6866e951\") "
Oct 30 12:49:36.182663 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.182202    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-kube-api-access-z92rf\" (UniqueName: \"kubernetes.io/projected/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-kube-api-access-z92rf\") pod \"2d7ffb87-9c58-44f7-a83a-ef0d6866e951\" (UID: \"2d7ffb87-9c58-44f7-a83a-ef0d6866e951\") "
Oct 30 12:49:36.182663 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.182443    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-maas-api-openshift-ca\") pod \"2d7ffb87-9c58-44f7-a83a-ef0d6866e951\" (UID: \"2d7ffb87-9c58-44f7-a83a-ef0d6866e951\") "
Oct 30 12:49:36.183448 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.183382    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-maas-api-openshift-ca" pod "2d7ffb87-9c58-44f7-a83a-ef0d6866e951" (UID: "2d7ffb87-9c58-44f7-a83a-ef0d6866e951"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:49:36.189145 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.189119    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-maas-api-xks-ca" pod "2d7ffb87-9c58-44f7-a83a-ef0d6866e951" (UID: "2d7ffb87-9c58-44f7-a83a-ef0d6866e951"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:49:36.189452 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.189430    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-kube-api-access-z92rf" pod "2d7ffb87-9c58-44f7-a83a-ef0d6866e951" (UID: "2d7ffb87-9c58-44f7-a83a-ef0d6866e951"). InnerVolumeSpecName "kube-api-access-z92rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:49:36.285007 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.284972    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z92rf\" (UniqueName: \"kubernetes.io/projected/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-kube-api-access-z92rf\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:36.285007 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.285002    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:36.285007 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.285011    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d7ffb87-9c58-44f7-a83a-ef0d6866e951-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:36.461780 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.461747    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"]
Oct 30 12:49:36.463638 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:49:36.463613    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af17d7f_a451_48d7_a03d_c25693a988f7.slice/crio-5a16e74e176a24df1ac646d408008d965cd4951347af937080959bb6ec49a35a WatchSource:0}: Error finding container 5a16e74e176a24df1ac646d408008d965cd4951347af937080959bb6ec49a35a: Status 404 returned error can't find the container with id 5a16e74e176a24df1ac646d408008d965cd4951347af937080959bb6ec49a35a
Oct 30 12:49:36.605144 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.605115    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"
Oct 30 12:49:36.605363 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.605115    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml" event={"ID":"2d7ffb87-9c58-44f7-a83a-ef0d6866e951","Type":"ContainerDied","Data":"9b76b7eed40bf5025c01f728d4eaece74e9bd8110fa3384d48f422fb5d4a90df"}
Oct 30 12:49:36.605363 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.605246    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b76b7eed40bf5025c01f728d4eaece74e9bd8110fa3384d48f422fb5d4a90df"
Oct 30 12:49:36.607601 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:36.607563    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt" event={"ID":"6af17d7f-a451-48d7-a03d-c25693a988f7","Type":"ContainerStarted","Data":"5a16e74e176a24df1ac646d408008d965cd4951347af937080959bb6ec49a35a"}
Oct 30 12:49:37.615798 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:37.615763    5837 generic.go:356] "Generic (PLEG): container finished" podID="6af17d7f-a451-48d7-a03d-c25693a988f7" containerID="79a0764d0b05f73db56629cc269696b9f236a18ef3d298057b035576fd7f14bf" exitCode=0
Oct 30 12:49:37.616267 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:37.615838    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt" event={"ID":"6af17d7f-a451-48d7-a03d-c25693a988f7","Type":"ContainerDied","Data":"79a0764d0b05f73db56629cc269696b9f236a18ef3d298057b035576fd7f14bf"}
Oct 30 12:49:38.778207 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:38.778181    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"
Oct 30 12:49:38.919006 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:38.918931    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af17d7f-a451-48d7-a03d-c25693a988f7-kube-api-access-7sk5s\" (UniqueName: \"kubernetes.io/projected/6af17d7f-a451-48d7-a03d-c25693a988f7-kube-api-access-7sk5s\") pod \"6af17d7f-a451-48d7-a03d-c25693a988f7\" (UID: \"6af17d7f-a451-48d7-a03d-c25693a988f7\") "
Oct 30 12:49:38.919161 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:38.919029    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af17d7f-a451-48d7-a03d-c25693a988f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af17d7f-a451-48d7-a03d-c25693a988f7-maas-api-xks-ca\") pod \"6af17d7f-a451-48d7-a03d-c25693a988f7\" (UID: \"6af17d7f-a451-48d7-a03d-c25693a988f7\") "
Oct 30 12:49:38.919161 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:38.919055    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af17d7f-a451-48d7-a03d-c25693a988f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af17d7f-a451-48d7-a03d-c25693a988f7-maas-api-openshift-ca\") pod \"6af17d7f-a451-48d7-a03d-c25693a988f7\" (UID: \"6af17d7f-a451-48d7-a03d-c25693a988f7\") "
Oct 30 12:49:38.919511 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:38.919474    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af17d7f-a451-48d7-a03d-c25693a988f7-maas-api-openshift-ca" pod "6af17d7f-a451-48d7-a03d-c25693a988f7" (UID: "6af17d7f-a451-48d7-a03d-c25693a988f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:49:38.921126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:38.921105    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af17d7f-a451-48d7-a03d-c25693a988f7-kube-api-access-7sk5s" pod "6af17d7f-a451-48d7-a03d-c25693a988f7" (UID: "6af17d7f-a451-48d7-a03d-c25693a988f7"). InnerVolumeSpecName "kube-api-access-7sk5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:49:38.921193 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:38.921161    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af17d7f-a451-48d7-a03d-c25693a988f7-maas-api-xks-ca" pod "6af17d7f-a451-48d7-a03d-c25693a988f7" (UID: "6af17d7f-a451-48d7-a03d-c25693a988f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:49:39.020872 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:39.020830    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sk5s\" (UniqueName: \"kubernetes.io/projected/6af17d7f-a451-48d7-a03d-c25693a988f7-kube-api-access-7sk5s\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:39.020872 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:39.020870    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af17d7f-a451-48d7-a03d-c25693a988f7-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:39.020872 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:39.020880    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af17d7f-a451-48d7-a03d-c25693a988f7-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:49:39.629954 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:39.629927    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"
Oct 30 12:49:39.630097 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:39.629959    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt" event={"ID":"6af17d7f-a451-48d7-a03d-c25693a988f7","Type":"ContainerDied","Data":"5a16e74e176a24df1ac646d408008d965cd4951347af937080959bb6ec49a35a"}
Oct 30 12:49:39.630097 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:39.630003    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a16e74e176a24df1ac646d408008d965cd4951347af937080959bb6ec49a35a"
Oct 30 12:49:41.683697 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.683666    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"]
Oct 30 12:49:41.684238 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.684225    5837 cpu_manager.go:414] "containerMap: removing container" podUID="2d7ffb87-9c58-44f7-a83a-ef0d6866e951" containerName="revoke-keys"
Oct 30 12:49:41.684295 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.684239    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7ffb87-9c58-44f7-a83a-ef0d6866e951" containerName="revoke-keys"
Oct 30 12:49:41.684295 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.684254    5837 cpu_manager.go:414] "containerMap: removing container" podUID="6af17d7f-a451-48d7-a03d-c25693a988f7" containerName="revoke-keys"
Oct 30 12:49:41.684295 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.684259    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af17d7f-a451-48d7-a03d-c25693a988f7" containerName="revoke-keys"
Oct 30 12:49:41.684387 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.684348    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7ffb87-9c58-44f7-a83a-ef0d6866e951" containerName="revoke-keys"
Oct 30 12:49:41.684387 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.684355    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af17d7f-a451-48d7-a03d-c25693a988f7" containerName="revoke-keys"
Oct 30 12:49:41.688758 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.688738    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.719065 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.719041    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"]
Oct 30 12:49:41.750669 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.750632    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.750844 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.750679    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.750844 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.750795    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.750974 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.750849    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8284bf7-b488-4b53-8804-4636dc743903-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.750974 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.750920    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxlj9\" (UniqueName: \"kubernetes.io/projected/f8284bf7-b488-4b53-8804-4636dc743903-kube-api-access-pxlj9\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.751084 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.751035    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.751141 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.751086    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8284bf7-b488-4b53-8804-4636dc743903-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.751225 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.751200    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.751287 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.751242    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8284bf7-b488-4b53-8804-4636dc743903-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.852835 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.852799    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.852990 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.852871    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8284bf7-b488-4b53-8804-4636dc743903-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.852990 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.852897    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxlj9\" (UniqueName: \"kubernetes.io/projected/f8284bf7-b488-4b53-8804-4636dc743903-kube-api-access-pxlj9\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.852990 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.852972    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.853190 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.853021    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8284bf7-b488-4b53-8804-4636dc743903-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.853190 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.853064    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.853190 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.853090    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8284bf7-b488-4b53-8804-4636dc743903-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.853342 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.853250    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.853342 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.853286    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.853480 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.853365    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.853558 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.853536    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.853669 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.853644    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.853790 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.853685    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.853790 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.853775    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8284bf7-b488-4b53-8804-4636dc743903-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.855472 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.855444    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.855670 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.855655    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8284bf7-b488-4b53-8804-4636dc743903-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.872205 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.872182    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8284bf7-b488-4b53-8804-4636dc743903-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:41.872420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:41.872383    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxlj9\" (UniqueName: \"kubernetes.io/projected/f8284bf7-b488-4b53-8804-4636dc743903-kube-api-access-pxlj9\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:42.003425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:42.003364    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:42.319388 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:42.319358    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"]
Oct 30 12:49:42.320718 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:49:42.320685    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice/crio-eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104 WatchSource:0}: Error finding container eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104: Status 404 returned error can't find the container with id eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104
Oct 30 12:49:42.323018 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:42.322985    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:42.323112 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:42.323060    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:42.323169 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:42.323107    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:42.652638 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:42.652562    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" event={"ID":"f8284bf7-b488-4b53-8804-4636dc743903","Type":"ContainerStarted","Data":"b228ce175ea14df44608df096c649b6efd65de9c4f3e0b595eac7d14405882fa"}
Oct 30 12:49:42.652638 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:42.652608    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" event={"ID":"f8284bf7-b488-4b53-8804-4636dc743903","Type":"ContainerStarted","Data":"eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104"}
Oct 30 12:49:43.008587 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:43.008550    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:43.010588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:43.010564    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 12:49:43.010702 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:43.010611    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 12:49:44.008419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:44.007796    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 12:49:44.008419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:44.007861    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 12:49:45.007419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:45.006813    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 12:49:45.007419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:45.006892    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 12:49:45.115930 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:49:45.115889    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:49:45.581659 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:45.581633    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:49:45.627775 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:45.627702    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" podStartSLOduration=4.627685926 podStartE2EDuration="4.627685926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:42.716706025 +0000 UTC m=+771.240990245" watchObservedRunningTime="2026-10-30 12:49:45.627685926 +0000 UTC m=+774.151970145"
Oct 30 12:49:46.008910 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.008873    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 12:49:46.009579 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.009538    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 12:49:46.344210 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.344132    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"]
Oct 30 12:49:46.348987 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.348964    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.366253 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.366230    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"]
Oct 30 12:49:46.371098 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.371075    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.379015 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.378988    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"]
Oct 30 12:49:46.420506 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.420466    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"]
Oct 30 12:49:46.535805 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.535773    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.535805 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.535807    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.536006 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.535836    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.536006 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.535930    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.536006 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.535965    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.536102 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.536055    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.536102 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.536090    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jv49\" (UniqueName: \"kubernetes.io/projected/faf87439-5044-43cd-9b1a-200188299f24-kube-api-access-4jv49\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.536162 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.536105    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.536197 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.536177    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.536247 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.536235    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.536285 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.536260    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.536285 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.536278    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xznpt\" (UniqueName: \"kubernetes.io/projected/f24ec435-0ad6-4388-8a13-f38ca9a26800-kube-api-access-xznpt\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.536361 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.536305    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf87439-5044-43cd-9b1a-200188299f24-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.536469 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.536452    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.536531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.536512    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf87439-5044-43cd-9b1a-200188299f24-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.536574 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.536561    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.536621 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.536588    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf87439-5044-43cd-9b1a-200188299f24-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.536621 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.536609    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24ec435-0ad6-4388-8a13-f38ca9a26800-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.638205 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.638127    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf87439-5044-43cd-9b1a-200188299f24-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.638205 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.638176    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.638424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.638216    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf87439-5044-43cd-9b1a-200188299f24-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.638424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.638241    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24ec435-0ad6-4388-8a13-f38ca9a26800-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.638424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.638369    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.638601 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.638434    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.638601 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.638471    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.638601 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.638501    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.638601 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.638554    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.638797 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.638623    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.638797 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.638672    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jv49\" (UniqueName: \"kubernetes.io/projected/faf87439-5044-43cd-9b1a-200188299f24-kube-api-access-4jv49\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.638907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.638790    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.638907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.638794    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.638907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.638831    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.639134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.639013    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.639134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.639102    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf87439-5044-43cd-9b1a-200188299f24-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.639134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.639124    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24ec435-0ad6-4388-8a13-f38ca9a26800-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.639260 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.639193    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.639305 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.639272    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.639373 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.639357    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.639439 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.639430    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.639473 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.639454    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.639512 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.639472    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xznpt\" (UniqueName: \"kubernetes.io/projected/f24ec435-0ad6-4388-8a13-f38ca9a26800-kube-api-access-xznpt\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.639551 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.639508    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf87439-5044-43cd-9b1a-200188299f24-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.639604 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.639556    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.639778 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.639759    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.639860 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.639799    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.639945 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.639928    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.641192 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.641167    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.641581 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.641557    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.642071 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.642048    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf87439-5044-43cd-9b1a-200188299f24-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.642175 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.642136    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.656995 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.656967    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jv49\" (UniqueName: \"kubernetes.io/projected/faf87439-5044-43cd-9b1a-200188299f24-kube-api-access-4jv49\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.657634 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.657612    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xznpt\" (UniqueName: \"kubernetes.io/projected/f24ec435-0ad6-4388-8a13-f38ca9a26800-kube-api-access-xznpt\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.658220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.658197    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:46.658220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.658209    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf87439-5044-43cd-9b1a-200188299f24-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.672225 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.672202    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:46.722912 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:46.722877    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:47.009426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.007773    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 12:49:47.009426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.007839    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 12:49:47.015351 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.015285    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"]
Oct 30 12:49:47.016571 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:49:47.016539    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf87439_5044_43cd_9b1a_200188299f24.slice/crio-3d7655000cbba8d1ad2f0f6202a60c14a731c641d71b91258fe9ab78ead0c175 WatchSource:0}: Error finding container 3d7655000cbba8d1ad2f0f6202a60c14a731c641d71b91258fe9ab78ead0c175: Status 404 returned error can't find the container with id 3d7655000cbba8d1ad2f0f6202a60c14a731c641d71b91258fe9ab78ead0c175
Oct 30 12:49:47.019476 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.019445    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:47.019558 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.019526    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:47.019607 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.019556    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:47.062606 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.062580    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"]
Oct 30 12:49:47.063013 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:49:47.062989    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24ec435_0ad6_4388_8a13_f38ca9a26800.slice/crio-e17f32c562b47ee8d6517da48863aa5ad44ab9b4e4e8966e99b39bf38a19d428 WatchSource:0}: Error finding container e17f32c562b47ee8d6517da48863aa5ad44ab9b4e4e8966e99b39bf38a19d428: Status 404 returned error can't find the container with id e17f32c562b47ee8d6517da48863aa5ad44ab9b4e4e8966e99b39bf38a19d428
Oct 30 12:49:47.065659 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.065628    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:47.065898 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.065704    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:47.065898 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.065745    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:47.730590 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.730548    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" event={"ID":"f24ec435-0ad6-4388-8a13-f38ca9a26800","Type":"ContainerStarted","Data":"3426311b73281d98ea110b5aea35377084bc5bd90ccc222b89c54d565cfadc54"}
Oct 30 12:49:47.732014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.731985    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" event={"ID":"f24ec435-0ad6-4388-8a13-f38ca9a26800","Type":"ContainerStarted","Data":"e17f32c562b47ee8d6517da48863aa5ad44ab9b4e4e8966e99b39bf38a19d428"}
Oct 30 12:49:47.739294 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.739262    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" event={"ID":"faf87439-5044-43cd-9b1a-200188299f24","Type":"ContainerStarted","Data":"45e9bf3f45778846fd02f494a46b8dfd062e6525e1c71dade9f7e3a24c1b3242"}
Oct 30 12:49:47.746681 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.746633    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" event={"ID":"faf87439-5044-43cd-9b1a-200188299f24","Type":"ContainerStarted","Data":"3d7655000cbba8d1ad2f0f6202a60c14a731c641d71b91258fe9ab78ead0c175"}
Oct 30 12:49:47.778629 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.776918    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" podStartSLOduration=1.7769000849999999 podStartE2EDuration="1.776900085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:47.775918035 +0000 UTC m=+776.300202250" watchObservedRunningTime="2026-10-30 12:49:47.776900085 +0000 UTC m=+776.301184305"
Oct 30 12:49:47.829734 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:47.829667    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" podStartSLOduration=1.8296443839999998 podStartE2EDuration="1.829644384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:47.82601701 +0000 UTC m=+776.350301328" watchObservedRunningTime="2026-10-30 12:49:47.829644384 +0000 UTC m=+776.353928604"
Oct 30 12:49:48.004960 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:48.004857    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 12:49:48.005229 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:48.005199    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 12:49:48.672687 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:48.672663    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:48.674533 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:48.674512    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 12:49:48.674641 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:48.674554    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" podUID="faf87439-5044-43cd-9b1a-200188299f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 12:49:48.730420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:48.729795    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:48.738456 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:48.731965    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 12:49:48.738456 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:48.732020    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 12:49:49.004416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:49.003849    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 12:49:49.004416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:49.003910    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 12:49:49.673212 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:49.672713    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 12:49:49.673212 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:49.672767    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" podUID="faf87439-5044-43cd-9b1a-200188299f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 12:49:49.728423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:49.728168    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 12:49:49.728423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:49.728227    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 12:49:50.018126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:50.018096    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:50.018941 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:50.018881    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:50.027694 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:50.027643    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 12:49:50.673078 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:50.673043    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 12:49:50.673268 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:50.673094    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" podUID="faf87439-5044-43cd-9b1a-200188299f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 12:49:50.729418 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:50.726797    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 12:49:50.729418 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:50.726854    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 12:49:51.673188 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:51.673154    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 12:49:51.673372 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:51.673202    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" podUID="faf87439-5044-43cd-9b1a-200188299f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 12:49:51.734838 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:51.734799    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 12:49:51.735036 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:51.734864    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 12:49:52.672814 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.672774    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 12:49:52.672991 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.672831    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" podUID="faf87439-5044-43cd-9b1a-200188299f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 12:49:52.729420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.728808    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 12:49:52.729420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.728913    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 12:49:52.924476 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.924033    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"]
Oct 30 12:49:52.935464 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.933669    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:52.944043 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.944009    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"]
Oct 30 12:49:52.951544 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.950815    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"]
Oct 30 12:49:52.951705 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.951568    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:52.972278 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.972181    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:52.972444 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.972302    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:52.972444 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.972343    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxzps\" (UniqueName: \"kubernetes.io/projected/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-kube-api-access-jxzps\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:52.972444 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.972421    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:52.972621 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.972446    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:52.972621 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.972604    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:52.972727 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.972654    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:52.972727 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.972678    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:52.972727 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.972704    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:52.974931 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:52.974906    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"]
Oct 30 12:49:53.074333 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.074300    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.074564 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.074385    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.074564 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.074472    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.074564 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.074495    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxzps\" (UniqueName: \"kubernetes.io/projected/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-kube-api-access-jxzps\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.074564 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.074533    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.074564 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.074556    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.074830 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.074607    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.074830 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.074657    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.074830 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.074683    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.074830 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.074713    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f855b07d-e71c-491e-b430-0d33bdd77a5d-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.075170 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.074834    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.075170 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.074866    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.075170 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.074890    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.075170 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.074975    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfdjp\" (UniqueName: \"kubernetes.io/projected/f855b07d-e71c-491e-b430-0d33bdd77a5d-kube-api-access-zfdjp\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.075446 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.075423    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.075548 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.075484    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.075548 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.075516    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.075736 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.075706    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.075798 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.075764    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.075798 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.075782    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.075911 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.075868    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.076035 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.076017    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.076170 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.076026    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.077865 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.077849    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.078875 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.078848    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.142124 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.142090    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxzps\" (UniqueName: \"kubernetes.io/projected/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-kube-api-access-jxzps\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.142275 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.142181    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.176680 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.176611    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.176680 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.176653    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.176680 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.176668    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.176869 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.176687    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f855b07d-e71c-491e-b430-0d33bdd77a5d-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.176869 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.176755    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.176869 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.176771    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.176869 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.176788    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfdjp\" (UniqueName: \"kubernetes.io/projected/f855b07d-e71c-491e-b430-0d33bdd77a5d-kube-api-access-zfdjp\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.176869 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.176820    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.176869 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.176857    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.177665 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.177296    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.177665 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.177635    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.177665 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.177643    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.177858 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.177645    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f855b07d-e71c-491e-b430-0d33bdd77a5d-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.177858 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.177774    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.179717 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.179697    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.179805 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.179790    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.235140 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.221696    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfdjp\" (UniqueName: \"kubernetes.io/projected/f855b07d-e71c-491e-b430-0d33bdd77a5d-kube-api-access-zfdjp\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.235140 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.234936    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.254235 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.254209    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:53.276530 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.276507    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:53.611372 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.611342    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"]
Oct 30 12:49:53.611792 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:49:53.611765    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9eaac99_ea1c_4fcb_b8e2_dc828e1089dd.slice/crio-6b3478d8ac67a0fbae0d0f95426a453f7e6237ff481b0e276cb825015e41aff0 WatchSource:0}: Error finding container 6b3478d8ac67a0fbae0d0f95426a453f7e6237ff481b0e276cb825015e41aff0: Status 404 returned error can't find the container with id 6b3478d8ac67a0fbae0d0f95426a453f7e6237ff481b0e276cb825015e41aff0
Oct 30 12:49:53.614576 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.614541    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:53.614657 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.614623    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:53.614696 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.614664    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:53.673437 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.673411    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 12:49:53.673556 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.673459    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" podUID="faf87439-5044-43cd-9b1a-200188299f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 12:49:53.700797 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.700772    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"]
Oct 30 12:49:53.702432 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:49:53.702359    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf855b07d_e71c_491e_b430_0d33bdd77a5d.slice/crio-01c698851232879aac3f55570099cbd649edf2e1afe9918a765e6f50c25c6b1d WatchSource:0}: Error finding container 01c698851232879aac3f55570099cbd649edf2e1afe9918a765e6f50c25c6b1d: Status 404 returned error can't find the container with id 01c698851232879aac3f55570099cbd649edf2e1afe9918a765e6f50c25c6b1d
Oct 30 12:49:53.704900 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.704869    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:53.704993 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.704930    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:53.704993 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.704961    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:49:53.724142 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.724028    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 12:49:53.724142 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.724090    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 12:49:53.792648 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.792547    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4" event={"ID":"f855b07d-e71c-491e-b430-0d33bdd77a5d","Type":"ContainerStarted","Data":"01c698851232879aac3f55570099cbd649edf2e1afe9918a765e6f50c25c6b1d"}
Oct 30 12:49:53.796200 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.795425    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk" event={"ID":"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd","Type":"ContainerStarted","Data":"113bd1bf944aff569dad5b15129f28b20dde65581aafc9c5a41aafda5b1b53a1"}
Oct 30 12:49:53.796200 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.795480    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk" event={"ID":"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd","Type":"ContainerStarted","Data":"6b3478d8ac67a0fbae0d0f95426a453f7e6237ff481b0e276cb825015e41aff0"}
Oct 30 12:49:53.842668 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:53.842227    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk" podStartSLOduration=1.8422076600000001 podStartE2EDuration="1.84220766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:53.836880437 +0000 UTC m=+782.361164659" watchObservedRunningTime="2026-10-30 12:49:53.84220766 +0000 UTC m=+782.366491883"
Oct 30 12:49:54.254959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:54.254924    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:49:54.256484 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:54.256454    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 12:49:54.256607 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:54.256510    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk" podUID="f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 12:49:54.679076 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:54.679001    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:54.679435 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:54.679419    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:54.680141 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:54.680125    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 12:49:54.751594 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:54.751561    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:54.753021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:54.752998    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:54.754072 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:54.754044    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 12:49:54.871980 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:54.871213    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4" event={"ID":"f855b07d-e71c-491e-b430-0d33bdd77a5d","Type":"ContainerStarted","Data":"370162a2b3317705827ce0b480b824b505697369b008d014d1ada2def086e1e5"}
Oct 30 12:49:54.967753 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:54.967631    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4" podStartSLOduration=2.967614337 podStartE2EDuration="2.967614337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:54.966272482 +0000 UTC m=+783.490556705" watchObservedRunningTime="2026-10-30 12:49:54.967614337 +0000 UTC m=+783.491898556"
Oct 30 12:49:55.255163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:55.255133    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 12:49:55.255437 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:55.255381    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk" podUID="f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 12:49:55.277555 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:55.277529    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:49:55.279260 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:55.279241    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 12:49:55.279338 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:55.279278    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4" podUID="f855b07d-e71c-491e-b430-0d33bdd77a5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 12:49:56.255610 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:56.255572    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 12:49:56.255994 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:56.255634    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk" podUID="f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 12:49:56.277566 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:56.277542    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 12:49:56.277698 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:56.277591    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4" podUID="f855b07d-e71c-491e-b430-0d33bdd77a5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 12:49:57.255411 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:57.255362    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 12:49:57.255606 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:57.255450    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk" podUID="f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 12:49:57.277833 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:57.277803    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 12:49:57.278188 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:57.277849    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4" podUID="f855b07d-e71c-491e-b430-0d33bdd77a5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 12:49:58.255310 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:58.255275    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 12:49:58.255525 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:58.255330    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk" podUID="f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 12:49:58.277818 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:58.277782    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 12:49:58.277948 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:58.277827    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4" podUID="f855b07d-e71c-491e-b430-0d33bdd77a5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 12:49:59.255348 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:59.255317    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 12:49:59.255554 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:59.255362    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk" podUID="f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 12:49:59.277331 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:59.277300    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 12:49:59.277583 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:49:59.277550    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4" podUID="f855b07d-e71c-491e-b430-0d33bdd77a5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 12:50:00.255088 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:00.255046    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 12:50:00.255611 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:00.255126    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk" podUID="f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 12:50:00.277254 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:00.277225    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 12:50:00.277388 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:00.277280    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4" podUID="f855b07d-e71c-491e-b430-0d33bdd77a5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 12:50:01.258713 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:01.258682    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:50:01.259156 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:01.259114    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:50:01.259694 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:01.259675    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 12:50:01.281308 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:01.281287    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:50:01.281620 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:01.281604    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:50:01.282314 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:01.282297    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 12:50:06.033343 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:06.033305    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:50:06.033931 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:06.033897    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:07.645905 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:07.645561    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:50:07.645905 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:07.645623    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:15.094690 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:50:15.094652    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:50:22.275699 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.275661    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"]
Oct 30 12:50:22.282872 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.282845    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:22.295964 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.295934    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"]
Oct 30 12:50:22.381053 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.381022    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-provider-credentials\") pod \"payload-processing-external-model-7dd7c9f6db-8qjsl\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") " pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:22.381198 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.381133    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h95k\" (UniqueName: \"kubernetes.io/projected/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-kube-api-access-2h95k\") pod \"payload-processing-external-model-7dd7c9f6db-8qjsl\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") " pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:22.381198 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.381188    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-routing-overlay-volume\") pod \"payload-processing-external-model-7dd7c9f6db-8qjsl\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") " pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:22.381274 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.381224    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-plugins-config-volume\") pod \"payload-processing-external-model-7dd7c9f6db-8qjsl\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") " pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:22.482797 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.482771    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-routing-overlay-volume\") pod \"payload-processing-external-model-7dd7c9f6db-8qjsl\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") " pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:22.482934 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.482814    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-plugins-config-volume\") pod \"payload-processing-external-model-7dd7c9f6db-8qjsl\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") " pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:22.482974 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.482939    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-provider-credentials\") pod \"payload-processing-external-model-7dd7c9f6db-8qjsl\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") " pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:22.483054 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.483010    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h95k\" (UniqueName: \"kubernetes.io/projected/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-kube-api-access-2h95k\") pod \"payload-processing-external-model-7dd7c9f6db-8qjsl\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") " pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:22.483545 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.483517    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-routing-overlay-volume\") pod \"payload-processing-external-model-7dd7c9f6db-8qjsl\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") " pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:22.483651 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.483560    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-plugins-config-volume\") pod \"payload-processing-external-model-7dd7c9f6db-8qjsl\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") " pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:22.485454 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.485439    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-provider-credentials\") pod \"payload-processing-external-model-7dd7c9f6db-8qjsl\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") " pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:22.500913 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.500888    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h95k\" (UniqueName: \"kubernetes.io/projected/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-kube-api-access-2h95k\") pod \"payload-processing-external-model-7dd7c9f6db-8qjsl\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") " pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:22.601260 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.601190    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:22.901786 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:22.901759    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"]
Oct 30 12:50:22.902872 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:50:22.902848    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78bb6dde_261f_4e4f_b2f1_22538ca48fe4.slice/crio-77c91c15a444b2a3004951e52b627be26663420f0d449abe6904c71f42593a48 WatchSource:0}: Error finding container 77c91c15a444b2a3004951e52b627be26663420f0d449abe6904c71f42593a48: Status 404 returned error can't find the container with id 77c91c15a444b2a3004951e52b627be26663420f0d449abe6904c71f42593a48
Oct 30 12:50:23.613115 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:23.613077    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl" event={"ID":"78bb6dde-261f-4e4f-b2f1-22538ca48fe4","Type":"ContainerStarted","Data":"c83540a30e1dd4b65e4e519d0ee8af9faf762836f3ecbdf19566e90f073db6e7"}
Oct 30 12:50:23.613533 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:23.613127    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl" event={"ID":"78bb6dde-261f-4e4f-b2f1-22538ca48fe4","Type":"ContainerStarted","Data":"77c91c15a444b2a3004951e52b627be26663420f0d449abe6904c71f42593a48"}
Oct 30 12:50:23.613533 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:23.613154    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:23.653166 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:23.653079    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl" podStartSLOduration=1.6530637929999998 podStartE2EDuration="1.653063793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:23.651813005 +0000 UTC m=+812.176097223" watchObservedRunningTime="2026-10-30 12:50:23.653063793 +0000 UTC m=+812.177348014"
Oct 30 12:50:30.148898 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.148862    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"]
Oct 30 12:50:30.155674 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.155650    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.182546 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.182514    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"]
Oct 30 12:50:30.189343 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.189318    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-105228\"/\"default-dockercfg-q265w\""
Oct 30 12:50:30.189516 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.189419    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-105228\"/\"openshift-service-ca.crt\""
Oct 30 12:50:30.191414 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.191339    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-105228\"/\"kube-root-ca.crt\""
Oct 30 12:50:30.191764 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.191745    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-105228\"/\"e2e-conflict-b-28de59c9-kserve-self-signed-certs\""
Oct 30 12:50:30.295722 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.295699    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab40783-0a3c-42c0-b9c3-b91f69b12417-tls-certs\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.295851 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.295732    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-tmp-dir\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.295851 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.295751    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-home\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.295976 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.295954    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nrmt\" (UniqueName: \"kubernetes.io/projected/aab40783-0a3c-42c0-b9c3-b91f69b12417-kube-api-access-8nrmt\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.296042 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.296027    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-dshm\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.296086 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.296048    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-model-cache\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.399134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.397433    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nrmt\" (UniqueName: \"kubernetes.io/projected/aab40783-0a3c-42c0-b9c3-b91f69b12417-kube-api-access-8nrmt\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.399134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.397556    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-dshm\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.399134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.397583    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-model-cache\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.399134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.397725    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab40783-0a3c-42c0-b9c3-b91f69b12417-tls-certs\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.399134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.397752    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-tmp-dir\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.399134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.397780    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-home\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.399134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.398327    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-home\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.410073 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.409754    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-dshm\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.410073 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.410032    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-model-cache\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.424426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.414191    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-tmp-dir\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.436911 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.435334    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab40783-0a3c-42c0-b9c3-b91f69b12417-tls-certs\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.442574 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.442485    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nrmt\" (UniqueName: \"kubernetes.io/projected/aab40783-0a3c-42c0-b9c3-b91f69b12417-kube-api-access-8nrmt\") pod \"e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") " pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.499919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.499888    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:30.810859 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:50:30.810824    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaab40783_0a3c_42c0_b9c3_b91f69b12417.slice/crio-1a317f4026cfdfe7314e48ef4e6b520dd7a79d8af5062f09484d406d41d8cb0e WatchSource:0}: Error finding container 1a317f4026cfdfe7314e48ef4e6b520dd7a79d8af5062f09484d406d41d8cb0e: Status 404 returned error can't find the container with id 1a317f4026cfdfe7314e48ef4e6b520dd7a79d8af5062f09484d406d41d8cb0e
Oct 30 12:50:30.811155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:30.811133    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"]
Oct 30 12:50:31.676418 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:31.672823    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs" event={"ID":"aab40783-0a3c-42c0-b9c3-b91f69b12417","Type":"ContainerStarted","Data":"bc7ab450b6b1ee2f833088e6445afaea09060708a762e0f24600da164e54d09f"}
Oct 30 12:50:31.676418 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:31.672897    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs" event={"ID":"aab40783-0a3c-42c0-b9c3-b91f69b12417","Type":"ContainerStarted","Data":"1a317f4026cfdfe7314e48ef4e6b520dd7a79d8af5062f09484d406d41d8cb0e"}
Oct 30 12:50:31.714458 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:31.714410    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs" podStartSLOduration=1.7143827900000002 podStartE2EDuration="1.71438279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:31.710406401 +0000 UTC m=+820.234690611" watchObservedRunningTime="2026-10-30 12:50:31.71438279 +0000 UTC m=+820.238667008"
Oct 30 12:50:32.500982 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:32.500945    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:34.649419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:34.647734    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:34.823426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:34.807992    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:50:34.823426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:34.808052    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" podUID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:35.424429 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:35.423623    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"]
Oct 30 12:50:40.424101 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.424043    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4" podUID="0c77ff27-6464-4cb8-875d-01a2dc4744fa" containerName="payload-processing" containerID="cri-o://f5261520e97e1cb38492d48f6f842906eb821aea472dfc75d95157ebbb2f8f1b" gracePeriod=25
Oct 30 12:50:40.500431 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.500382    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:40.517947 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.517914    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:40.704734 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.704706    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:50:40.792152 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.792125    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c77ff27-6464-4cb8-875d-01a2dc4744fa-kube-api-access-7wtwt\" (UniqueName: \"kubernetes.io/projected/0c77ff27-6464-4cb8-875d-01a2dc4744fa-kube-api-access-7wtwt\") pod \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") "
Oct 30 12:50:40.792322 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.792161    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c77ff27-6464-4cb8-875d-01a2dc4744fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c77ff27-6464-4cb8-875d-01a2dc4744fa-plugins-config-volume\") pod \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") "
Oct 30 12:50:40.792322 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.792260    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c77ff27-6464-4cb8-875d-01a2dc4744fa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c77ff27-6464-4cb8-875d-01a2dc4744fa-provider-credentials\") pod \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") "
Oct 30 12:50:40.792469 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.792345    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c77ff27-6464-4cb8-875d-01a2dc4744fa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c77ff27-6464-4cb8-875d-01a2dc4744fa-routing-overlay-volume\") pod \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\" (UID: \"0c77ff27-6464-4cb8-875d-01a2dc4744fa\") "
Oct 30 12:50:40.792677 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.792649    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c77ff27-6464-4cb8-875d-01a2dc4744fa-plugins-config-volume" pod "0c77ff27-6464-4cb8-875d-01a2dc4744fa" (UID: "0c77ff27-6464-4cb8-875d-01a2dc4744fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:50:40.793018 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.792992    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c77ff27-6464-4cb8-875d-01a2dc4744fa-routing-overlay-volume" pod "0c77ff27-6464-4cb8-875d-01a2dc4744fa" (UID: "0c77ff27-6464-4cb8-875d-01a2dc4744fa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:50:40.793247 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.793094    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c77ff27-6464-4cb8-875d-01a2dc4744fa-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:50:40.793247 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.793115    5837 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c77ff27-6464-4cb8-875d-01a2dc4744fa-routing-overlay-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:50:40.794834 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.794806    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c77ff27-6464-4cb8-875d-01a2dc4744fa-provider-credentials" pod "0c77ff27-6464-4cb8-875d-01a2dc4744fa" (UID: "0c77ff27-6464-4cb8-875d-01a2dc4744fa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:50:40.794976 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.794960    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c77ff27-6464-4cb8-875d-01a2dc4744fa-kube-api-access-7wtwt" pod "0c77ff27-6464-4cb8-875d-01a2dc4744fa" (UID: "0c77ff27-6464-4cb8-875d-01a2dc4744fa"). InnerVolumeSpecName "kube-api-access-7wtwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:50:40.894090 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.894062    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wtwt\" (UniqueName: \"kubernetes.io/projected/0c77ff27-6464-4cb8-875d-01a2dc4744fa-kube-api-access-7wtwt\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:50:40.894090 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:40.894088    5837 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c77ff27-6464-4cb8-875d-01a2dc4744fa-provider-credentials\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:50:41.099953 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:41.099924    5837 generic.go:356] "Generic (PLEG): container finished" podID="0c77ff27-6464-4cb8-875d-01a2dc4744fa" containerID="f5261520e97e1cb38492d48f6f842906eb821aea472dfc75d95157ebbb2f8f1b" exitCode=0
Oct 30 12:50:41.100146 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:41.099986    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"
Oct 30 12:50:41.100146 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:41.100001    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4" event={"ID":"0c77ff27-6464-4cb8-875d-01a2dc4744fa","Type":"ContainerDied","Data":"f5261520e97e1cb38492d48f6f842906eb821aea472dfc75d95157ebbb2f8f1b"}
Oct 30 12:50:41.100146 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:41.100055    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4" event={"ID":"0c77ff27-6464-4cb8-875d-01a2dc4744fa","Type":"ContainerDied","Data":"84a84010445af6f22f2f7ac11c896676e1e3663f47952baff0576f833a32ad73"}
Oct 30 12:50:41.100146 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:41.100067    5837 scope.go:122] "RemoveContainer" containerID="f5261520e97e1cb38492d48f6f842906eb821aea472dfc75d95157ebbb2f8f1b"
Oct 30 12:50:41.113117 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:41.113089    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:50:41.121744 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:41.118515    5837 scope.go:122] "RemoveContainer" containerID="f5261520e97e1cb38492d48f6f842906eb821aea472dfc75d95157ebbb2f8f1b"
Oct 30 12:50:41.121744 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:50:41.120221    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5261520e97e1cb38492d48f6f842906eb821aea472dfc75d95157ebbb2f8f1b\": container with ID starting with f5261520e97e1cb38492d48f6f842906eb821aea472dfc75d95157ebbb2f8f1b not found: ID does not exist" containerID="f5261520e97e1cb38492d48f6f842906eb821aea472dfc75d95157ebbb2f8f1b"
Oct 30 12:50:41.121744 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:41.120254    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5261520e97e1cb38492d48f6f842906eb821aea472dfc75d95157ebbb2f8f1b"} err="failed to get container status \"f5261520e97e1cb38492d48f6f842906eb821aea472dfc75d95157ebbb2f8f1b\": rpc error: code = NotFound desc = could not find container \"f5261520e97e1cb38492d48f6f842906eb821aea472dfc75d95157ebbb2f8f1b\": container with ID starting with f5261520e97e1cb38492d48f6f842906eb821aea472dfc75d95157ebbb2f8f1b not found: ID does not exist"
Oct 30 12:50:41.151221 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:41.151196    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"]
Oct 30 12:50:41.162518 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:41.162496    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c9d5b9b75-tpcg4"]
Oct 30 12:50:41.605739 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:41.605704    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c77ff27-6464-4cb8-875d-01a2dc4744fa" path="/var/lib/kubelet/pods/0c77ff27-6464-4cb8-875d-01a2dc4744fa/volumes"
Oct 30 12:50:45.094304 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:50:45.094269    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:50:52.649329 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:52.649293    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"]
Oct 30 12:50:53.228296 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.228264    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"]
Oct 30 12:50:53.228976 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.228959    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0c77ff27-6464-4cb8-875d-01a2dc4744fa" containerName="payload-processing"
Oct 30 12:50:53.229024 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.228978    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c77ff27-6464-4cb8-875d-01a2dc4744fa" containerName="payload-processing"
Oct 30 12:50:53.229092 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.229083    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c77ff27-6464-4cb8-875d-01a2dc4744fa" containerName="payload-processing"
Oct 30 12:50:53.233911 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.233891    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.263598 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.263568    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"]
Oct 30 12:50:53.355970 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.355934    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.356131 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.355998    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.356131 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.356103    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.356205 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.356163    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.356205 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.356186    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.356205 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.356203    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.356368 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.356355    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.356430 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.356415    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.356492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.356475    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dtql\" (UniqueName: \"kubernetes.io/projected/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-kube-api-access-8dtql\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.457940 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.457896    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.457940 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.457944    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.458301 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.457972    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.458301 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.457995    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.458301 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.458012    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.458301 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.458094    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.458301 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.458139    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.458301 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.458189    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dtql\" (UniqueName: \"kubernetes.io/projected/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-kube-api-access-8dtql\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.458648 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.458306    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.458703 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.458666    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.458763 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.458716    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.458763 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.458726    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.458848 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.458767    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.458848 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.458785    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.460606 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.460585    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.460957 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.460933    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.477208 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.477177    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.477853 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.477836    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dtql\" (UniqueName: \"kubernetes.io/projected/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-kube-api-access-8dtql\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.547888 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.547862    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:53.900838 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:50:53.900807    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ea72c6_dacd_42c2_a7c1_c32ea4135e66.slice/crio-a67dffe96879eb386c71bee5d7e09fe06dc45faf35898c6db76a9c5a4b164c30 WatchSource:0}: Error finding container a67dffe96879eb386c71bee5d7e09fe06dc45faf35898c6db76a9c5a4b164c30: Status 404 returned error can't find the container with id a67dffe96879eb386c71bee5d7e09fe06dc45faf35898c6db76a9c5a4b164c30
Oct 30 12:50:53.901237 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.900894    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"]
Oct 30 12:50:53.903227 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.903197    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:50:53.903312 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.903261    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:50:53.903312 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:53.903288    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:50:54.195980 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:54.195873    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" event={"ID":"80ea72c6-dacd-42c2-a7c1-c32ea4135e66","Type":"ContainerStarted","Data":"37bd9503cb445a8d6d04c627d02af8b0cdbd18b892f0f9f4d83aeb74011f8f9b"}
Oct 30 12:50:54.195980 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:54.195941    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" event={"ID":"80ea72c6-dacd-42c2-a7c1-c32ea4135e66","Type":"ContainerStarted","Data":"a67dffe96879eb386c71bee5d7e09fe06dc45faf35898c6db76a9c5a4b164c30"}
Oct 30 12:50:54.240698 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:54.240648    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podStartSLOduration=1.240632395 podStartE2EDuration="1.240632395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:54.23463485 +0000 UTC m=+842.758919070" watchObservedRunningTime="2026-10-30 12:50:54.240632395 +0000 UTC m=+842.764916657"
Oct 30 12:50:54.547991 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:54.547955    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:50:54.549856 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:54.549828    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 12:50:54.550000 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:54.549883    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 12:50:55.548641 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:55.548598    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 12:50:55.549151 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:55.548651    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 12:50:56.548599 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:56.548566    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 12:50:56.548799 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:56.548624    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 12:50:57.549051 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:57.549021    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 12:50:57.549451 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:57.549071    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 12:50:57.649808 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:57.649774    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl" podUID="78bb6dde-261f-4e4f-b2f1-22538ca48fe4" containerName="payload-processing" containerID="cri-o://c83540a30e1dd4b65e4e519d0ee8af9faf762836f3ecbdf19566e90f073db6e7" gracePeriod=25
Oct 30 12:50:57.952900 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:57.952868    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:58.044210 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.044164    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-provider-credentials\") pod \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") "
Oct 30 12:50:58.044435 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.044319    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-routing-overlay-volume\") pod \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") "
Oct 30 12:50:58.044435 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.044380    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-kube-api-access-2h95k\" (UniqueName: \"kubernetes.io/projected/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-kube-api-access-2h95k\") pod \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") "
Oct 30 12:50:58.044586 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.044499    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-plugins-config-volume\") pod \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\" (UID: \"78bb6dde-261f-4e4f-b2f1-22538ca48fe4\") "
Oct 30 12:50:58.044848 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.044819    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-routing-overlay-volume" pod "78bb6dde-261f-4e4f-b2f1-22538ca48fe4" (UID: "78bb6dde-261f-4e4f-b2f1-22538ca48fe4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:50:58.045096 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.045073    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-plugins-config-volume" pod "78bb6dde-261f-4e4f-b2f1-22538ca48fe4" (UID: "78bb6dde-261f-4e4f-b2f1-22538ca48fe4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:50:58.045220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.045201    5837 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-routing-overlay-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:50:58.045283 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.045225    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:50:58.047418 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.047123    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-provider-credentials" pod "78bb6dde-261f-4e4f-b2f1-22538ca48fe4" (UID: "78bb6dde-261f-4e4f-b2f1-22538ca48fe4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:50:58.047418 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.047208    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-kube-api-access-2h95k" pod "78bb6dde-261f-4e4f-b2f1-22538ca48fe4" (UID: "78bb6dde-261f-4e4f-b2f1-22538ca48fe4"). InnerVolumeSpecName "kube-api-access-2h95k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:50:58.146327 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.146249    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h95k\" (UniqueName: \"kubernetes.io/projected/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-kube-api-access-2h95k\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:50:58.146327 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.146278    5837 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78bb6dde-261f-4e4f-b2f1-22538ca48fe4-provider-credentials\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:50:58.243161 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.243128    5837 generic.go:356] "Generic (PLEG): container finished" podID="78bb6dde-261f-4e4f-b2f1-22538ca48fe4" containerID="c83540a30e1dd4b65e4e519d0ee8af9faf762836f3ecbdf19566e90f073db6e7" exitCode=0
Oct 30 12:50:58.243321 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.243176    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"
Oct 30 12:50:58.243321 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.243180    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl" event={"ID":"78bb6dde-261f-4e4f-b2f1-22538ca48fe4","Type":"ContainerDied","Data":"c83540a30e1dd4b65e4e519d0ee8af9faf762836f3ecbdf19566e90f073db6e7"}
Oct 30 12:50:58.243321 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.243296    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl" event={"ID":"78bb6dde-261f-4e4f-b2f1-22538ca48fe4","Type":"ContainerDied","Data":"77c91c15a444b2a3004951e52b627be26663420f0d449abe6904c71f42593a48"}
Oct 30 12:50:58.243472 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.243326    5837 scope.go:122] "RemoveContainer" containerID="c83540a30e1dd4b65e4e519d0ee8af9faf762836f3ecbdf19566e90f073db6e7"
Oct 30 12:50:58.254844 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.254821    5837 scope.go:122] "RemoveContainer" containerID="c83540a30e1dd4b65e4e519d0ee8af9faf762836f3ecbdf19566e90f073db6e7"
Oct 30 12:50:58.255185 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:50:58.255162    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c83540a30e1dd4b65e4e519d0ee8af9faf762836f3ecbdf19566e90f073db6e7\": container with ID starting with c83540a30e1dd4b65e4e519d0ee8af9faf762836f3ecbdf19566e90f073db6e7 not found: ID does not exist" containerID="c83540a30e1dd4b65e4e519d0ee8af9faf762836f3ecbdf19566e90f073db6e7"
Oct 30 12:50:58.255281 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.255196    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c83540a30e1dd4b65e4e519d0ee8af9faf762836f3ecbdf19566e90f073db6e7"} err="failed to get container status \"c83540a30e1dd4b65e4e519d0ee8af9faf762836f3ecbdf19566e90f073db6e7\": rpc error: code = NotFound desc = could not find container \"c83540a30e1dd4b65e4e519d0ee8af9faf762836f3ecbdf19566e90f073db6e7\": container with ID starting with c83540a30e1dd4b65e4e519d0ee8af9faf762836f3ecbdf19566e90f073db6e7 not found: ID does not exist"
Oct 30 12:50:58.314329 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.313145    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"]
Oct 30 12:50:58.317073 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.317047    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7dd7c9f6db-8qjsl"]
Oct 30 12:50:58.549024 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.548991    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 12:50:58.549225 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:58.549053    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 12:50:59.551360 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:59.550772    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 12:50:59.551360 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:59.550838    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 12:50:59.742468 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:50:59.738950    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78bb6dde-261f-4e4f-b2f1-22538ca48fe4" path="/var/lib/kubelet/pods/78bb6dde-261f-4e4f-b2f1-22538ca48fe4/volumes"
Oct 30 12:51:00.550787 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:00.549788    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 12:51:00.550787 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:00.549863    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 12:51:01.780903 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:01.780507    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:01.780903 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:01.780592    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" podUID="48deaa90-e66d-454f-af97-1a5462424e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:01.819007 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:01.818973    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:01.819262 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:01.819239    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:02.040467 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.036989    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"]
Oct 30 12:51:02.040467 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.037920    5837 cpu_manager.go:414] "containerMap: removing container" podUID="78bb6dde-261f-4e4f-b2f1-22538ca48fe4" containerName="payload-processing"
Oct 30 12:51:02.040467 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.037939    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78bb6dde-261f-4e4f-b2f1-22538ca48fe4" containerName="payload-processing"
Oct 30 12:51:02.040467 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.038068    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="78bb6dde-261f-4e4f-b2f1-22538ca48fe4" containerName="payload-processing"
Oct 30 12:51:02.079843 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.079812    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.274428 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.264466    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4cd3fb\"/\"openshift-service-ca.crt\""
Oct 30 12:51:02.274428 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.264678    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4cd3fb\"/\"default-dockercfg-slm6x\""
Oct 30 12:51:02.274428 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.264805    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4cd3fb\"/\"distinct-2-w0-4cd3fb-kserve-self-signed-certs\""
Oct 30 12:51:02.274428 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.264907    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4cd3fb\"/\"kube-root-ca.crt\""
Oct 30 12:51:02.431478 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.389946    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-tls-certs\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.431478 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.390046    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-dshm\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.431478 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.390072    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-tmp-dir\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.431478 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.390207    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-home\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.431478 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.390265    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-model-cache\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.431478 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.390291    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngb5w\" (UniqueName: \"kubernetes.io/projected/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-kube-api-access-ngb5w\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.465446 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.465409    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"]
Oct 30 12:51:02.573933 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.573892    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-tls-certs\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.689165 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.688899    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-tls-certs\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.689912 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.689875    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-dshm\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.690126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.690107    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-tmp-dir\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.708679 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.708654    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-home\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.917037 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.690606    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-tmp-dir\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.917655 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.719067    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-home\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.917749 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.837253    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-dshm\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.917954 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.917931    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-model-cache\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.927536 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.927493    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngb5w\" (UniqueName: \"kubernetes.io/projected/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-kube-api-access-ngb5w\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:02.940516 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:02.918346    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-model-cache\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:03.071419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:03.046641    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:03.072078 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:03.072018    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:03.113271 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:03.113241    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngb5w\" (UniqueName: \"kubernetes.io/projected/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-kube-api-access-ngb5w\") pod \"distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:03.211593 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:03.211491    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs" podUID="aab40783-0a3c-42c0-b9c3-b91f69b12417" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:03.414424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:03.413647    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:04.102741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:04.102650    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:04.103291 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:04.103259    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:04.331624 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:04.331566    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs" podUID="aab40783-0a3c-42c0-b9c3-b91f69b12417" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:04.332592 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:04.332569    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:51:05.194911 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:05.194857    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:05.195589 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:05.195555    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:05.368416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:05.353557    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs" podUID="aab40783-0a3c-42c0-b9c3-b91f69b12417" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:05.395414 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:05.391997    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"]
Oct 30 12:51:05.497741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:05.496725    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:51:06.117472 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:06.109020    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:06.117472 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:06.109095    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:06.252421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:06.247743    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:06.252421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:06.247808    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:06.321937 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:06.321901    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" event={"ID":"9ab4a57e-f122-4801-8bee-3fc3f6d550f3","Type":"ContainerStarted","Data":"79e41aab6c1322aa55874f2859d71ec5936e224097a3c5d40f6f10a23be24ab3"}
Oct 30 12:51:06.323251 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:06.323227    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" event={"ID":"9ab4a57e-f122-4801-8bee-3fc3f6d550f3","Type":"ContainerStarted","Data":"7b9e0ff355579b506d078b7dc86660e581118bfcd09b1af9589d8a256b93bc63"}
Oct 30 12:51:06.356515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:06.356481    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:06.356771 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:06.356741    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:06.370517 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:06.370417    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podStartSLOduration=5.370374176 podStartE2EDuration="5.370374176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:06.37027567 +0000 UTC m=+854.894559891" watchObservedRunningTime="2026-10-30 12:51:06.370374176 +0000 UTC m=+854.894658392"
Oct 30 12:51:06.419572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:06.419540    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:06.476292 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:06.476112    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:51:06.479446 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:06.478801    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:51:06.480303 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:06.480058    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 12:51:07.088109 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:07.088072    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"]
Oct 30 12:51:13.419365 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:13.419336    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:13.432073 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:13.432049    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:14.390576 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:14.390545    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:51:15.094872 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:51:15.094835    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:51:22.112806 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.112757    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs" podUID="aab40783-0a3c-42c0-b9c3-b91f69b12417" containerName="main" containerID="cri-o://bc7ab450b6b1ee2f833088e6445afaea09060708a762e0f24600da164e54d09f" gracePeriod=45
Oct 30 12:51:22.418808 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.418787    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:51:22.440165 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.440141    5837 generic.go:356] "Generic (PLEG): container finished" podID="aab40783-0a3c-42c0-b9c3-b91f69b12417" containerID="bc7ab450b6b1ee2f833088e6445afaea09060708a762e0f24600da164e54d09f" exitCode=0
Oct 30 12:51:22.440315 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.440185    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"
Oct 30 12:51:22.440315 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.440221    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs" event={"ID":"aab40783-0a3c-42c0-b9c3-b91f69b12417","Type":"ContainerDied","Data":"bc7ab450b6b1ee2f833088e6445afaea09060708a762e0f24600da164e54d09f"}
Oct 30 12:51:22.440315 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.440263    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs" event={"ID":"aab40783-0a3c-42c0-b9c3-b91f69b12417","Type":"ContainerDied","Data":"1a317f4026cfdfe7314e48ef4e6b520dd7a79d8af5062f09484d406d41d8cb0e"}
Oct 30 12:51:22.440315 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.440279    5837 scope.go:122] "RemoveContainer" containerID="bc7ab450b6b1ee2f833088e6445afaea09060708a762e0f24600da164e54d09f"
Oct 30 12:51:22.450425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.450375    5837 scope.go:122] "RemoveContainer" containerID="bc7ab450b6b1ee2f833088e6445afaea09060708a762e0f24600da164e54d09f"
Oct 30 12:51:22.450718 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:51:22.450699    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7ab450b6b1ee2f833088e6445afaea09060708a762e0f24600da164e54d09f\": container with ID starting with bc7ab450b6b1ee2f833088e6445afaea09060708a762e0f24600da164e54d09f not found: ID does not exist" containerID="bc7ab450b6b1ee2f833088e6445afaea09060708a762e0f24600da164e54d09f"
Oct 30 12:51:22.450772 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.450727    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7ab450b6b1ee2f833088e6445afaea09060708a762e0f24600da164e54d09f"} err="failed to get container status \"bc7ab450b6b1ee2f833088e6445afaea09060708a762e0f24600da164e54d09f\": rpc error: code = NotFound desc = could not find container \"bc7ab450b6b1ee2f833088e6445afaea09060708a762e0f24600da164e54d09f\": container with ID starting with bc7ab450b6b1ee2f833088e6445afaea09060708a762e0f24600da164e54d09f not found: ID does not exist"
Oct 30 12:51:22.472575 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.470648    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-model-cache\") pod \"aab40783-0a3c-42c0-b9c3-b91f69b12417\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") "
Oct 30 12:51:22.472575 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.470714    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-home\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-home\") pod \"aab40783-0a3c-42c0-b9c3-b91f69b12417\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") "
Oct 30 12:51:22.472575 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.470780    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab40783-0a3c-42c0-b9c3-b91f69b12417-kube-api-access-8nrmt\" (UniqueName: \"kubernetes.io/projected/aab40783-0a3c-42c0-b9c3-b91f69b12417-kube-api-access-8nrmt\") pod \"aab40783-0a3c-42c0-b9c3-b91f69b12417\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") "
Oct 30 12:51:22.472575 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.470944    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-dshm\") pod \"aab40783-0a3c-42c0-b9c3-b91f69b12417\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") "
Oct 30 12:51:22.472575 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.470988    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-tmp-dir\") pod \"aab40783-0a3c-42c0-b9c3-b91f69b12417\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") "
Oct 30 12:51:22.472575 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.471141    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aab40783-0a3c-42c0-b9c3-b91f69b12417-tls-certs\" (UniqueName: \"kubernetes.io/secret/aab40783-0a3c-42c0-b9c3-b91f69b12417-tls-certs\") pod \"aab40783-0a3c-42c0-b9c3-b91f69b12417\" (UID: \"aab40783-0a3c-42c0-b9c3-b91f69b12417\") "
Oct 30 12:51:22.472575 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.471142    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-model-cache" pod "aab40783-0a3c-42c0-b9c3-b91f69b12417" (UID: "aab40783-0a3c-42c0-b9c3-b91f69b12417"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:51:22.472575 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.471341    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-home" pod "aab40783-0a3c-42c0-b9c3-b91f69b12417" (UID: "aab40783-0a3c-42c0-b9c3-b91f69b12417"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:51:22.472575 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.471938    5837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-model-cache\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:51:22.472575 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.471965    5837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-home\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:51:22.473465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.473434    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-tmp-dir" pod "aab40783-0a3c-42c0-b9c3-b91f69b12417" (UID: "aab40783-0a3c-42c0-b9c3-b91f69b12417"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:51:22.474304 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.474276    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-dshm" pod "aab40783-0a3c-42c0-b9c3-b91f69b12417" (UID: "aab40783-0a3c-42c0-b9c3-b91f69b12417"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:51:22.474805 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.474780    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aab40783-0a3c-42c0-b9c3-b91f69b12417-tls-certs" pod "aab40783-0a3c-42c0-b9c3-b91f69b12417" (UID: "aab40783-0a3c-42c0-b9c3-b91f69b12417"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:51:22.475774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.475746    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab40783-0a3c-42c0-b9c3-b91f69b12417-kube-api-access-8nrmt" pod "aab40783-0a3c-42c0-b9c3-b91f69b12417" (UID: "aab40783-0a3c-42c0-b9c3-b91f69b12417"). InnerVolumeSpecName "kube-api-access-8nrmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:51:22.573348 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.573319    5837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab40783-0a3c-42c0-b9c3-b91f69b12417-tls-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:51:22.573348 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.573348    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nrmt\" (UniqueName: \"kubernetes.io/projected/aab40783-0a3c-42c0-b9c3-b91f69b12417-kube-api-access-8nrmt\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:51:22.573559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.573362    5837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-dshm\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:51:22.573559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.573373    5837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab40783-0a3c-42c0-b9c3-b91f69b12417-tmp-dir\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:51:22.784008 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.783977    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"]
Oct 30 12:51:22.789688 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:22.789662    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs"]
Oct 30 12:51:23.371822 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:23.371783    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/e2e-conflict-b-28de59c9-kserve-6d6f79c6d-xzjrs" podUID="aab40783-0a3c-42c0-b9c3-b91f69b12417" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:23.605843 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:23.605811    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aab40783-0a3c-42c0-b9c3-b91f69b12417" path="/var/lib/kubelet/pods/aab40783-0a3c-42c0-b9c3-b91f69b12417/volumes"
Oct 30 12:51:29.069222 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.062200    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"]
Oct 30 12:51:29.076982 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.070922    5837 cpu_manager.go:414] "containerMap: removing container" podUID="aab40783-0a3c-42c0-b9c3-b91f69b12417" containerName="main"
Oct 30 12:51:29.077223 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.077199    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab40783-0a3c-42c0-b9c3-b91f69b12417" containerName="main"
Oct 30 12:51:29.077627 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.077605    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab40783-0a3c-42c0-b9c3-b91f69b12417" containerName="main"
Oct 30 12:51:29.126427 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.118859    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"]
Oct 30 12:51:29.126427 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.119079    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.145419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.139812    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7a590a\"/\"kube-root-ca.crt\""
Oct 30 12:51:29.189421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.187447    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7a590a\"/\"distinct-2-w1-7a590a-kserve-self-signed-certs\""
Oct 30 12:51:29.189421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.187531    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7a590a\"/\"openshift-service-ca.crt\""
Oct 30 12:51:29.189421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.187586    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7a590a\"/\"default-dockercfg-lzwt6\""
Oct 30 12:51:29.194433 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.194403    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6eaa6ea-8274-49fa-b007-753de046e268-tls-certs\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.195086 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.195062    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt5lj\" (UniqueName: \"kubernetes.io/projected/d6eaa6ea-8274-49fa-b007-753de046e268-kube-api-access-lt5lj\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.196475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.196452    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-tmp-dir\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.199821 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.196860    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-dshm\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.199821 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.199727    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-model-cache\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.199991 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.199917    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-home\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.308126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.306628    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-home\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.308126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.306798    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6eaa6ea-8274-49fa-b007-753de046e268-tls-certs\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.308126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.306824    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt5lj\" (UniqueName: \"kubernetes.io/projected/d6eaa6ea-8274-49fa-b007-753de046e268-kube-api-access-lt5lj\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.308126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.306882    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-tmp-dir\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.308126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.306934    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-dshm\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.308126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.306957    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-model-cache\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.308126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.307505    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-model-cache\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.308126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.307753    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-home\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.329597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.317077    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-tmp-dir\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.364019 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.363736    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-dshm\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.375858 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.375792    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt5lj\" (UniqueName: \"kubernetes.io/projected/d6eaa6ea-8274-49fa-b007-753de046e268-kube-api-access-lt5lj\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.410422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.402096    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6eaa6ea-8274-49fa-b007-753de046e268-tls-certs\") pod \"distinct-2-w1-7a590a-kserve-9d486856b-nx762\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") " pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:29.446723 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:29.445819    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:31.107953 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:31.107918    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"]
Oct 30 12:51:31.162529 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:51:31.162418    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6eaa6ea_8274_49fa_b007_753de046e268.slice/crio-82c8aff0a04d9661da73c494f6d62824a4260f499e42c30ca89938a0d7637918\": RecentStats: unable to find data in memory cache]"
Oct 30 12:51:31.570415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:31.567010    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:51:31.786733 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:31.786672    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:31.787141 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:31.787101    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" podUID="48deaa90-e66d-454f-af97-1a5462424e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:32.002095 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:32.002057    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:32.002445 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:32.002410    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:32.402420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:32.401500    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:32.402420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:32.401584    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:32.565444 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:32.536776    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" event={"ID":"d6eaa6ea-8274-49fa-b007-753de046e268","Type":"ContainerStarted","Data":"82c8aff0a04d9661da73c494f6d62824a4260f499e42c30ca89938a0d7637918"}
Oct 30 12:51:33.493422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:33.483516    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:51:34.778419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:34.752753    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" event={"ID":"d6eaa6ea-8274-49fa-b007-753de046e268","Type":"ContainerStarted","Data":"01141dc5895af7f8abb7e47c9a0fdecd21088727db28a51324ede210d5af9281"}
Oct 30 12:51:34.921419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:34.900990    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podStartSLOduration=5.900967468 podStartE2EDuration="5.900967468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:34.810416638 +0000 UTC m=+883.334700852" watchObservedRunningTime="2026-10-30 12:51:34.900967468 +0000 UTC m=+883.425251687"
Oct 30 12:51:35.447774 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:35.447742    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:39.447195 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:39.447161    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:39.459724 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:39.459701    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:39.806561 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:39.806533    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:51:45.133383 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:51:45.133322    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:51:49.471990 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.471959    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"]
Oct 30 12:51:49.476250 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.476231    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"
Oct 30 12:51:49.497261 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.497236    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"]
Oct 30 12:51:49.503665 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.503641    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:51:49.503795 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.503648    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:51:49.503874 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.503660    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:51:49.503874 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.503704    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:51:49.575931 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.575903    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc947c90-b4bc-4fc0-9935-10706af098cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-152b8586-d747j\" (UID: \"bc947c90-b4bc-4fc0-9935-10706af098cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"
Oct 30 12:51:49.576086 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.576064    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc947c90-b4bc-4fc0-9935-10706af098cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-152b8586-d747j\" (UID: \"bc947c90-b4bc-4fc0-9935-10706af098cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"
Oct 30 12:51:49.576172 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.576159    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk6xj\" (UniqueName: \"kubernetes.io/projected/bc947c90-b4bc-4fc0-9935-10706af098cd-kube-api-access-hk6xj\") pod \"maas-api-revoke-sub-152b8586-d747j\" (UID: \"bc947c90-b4bc-4fc0-9935-10706af098cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"
Oct 30 12:51:49.677581 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.677553    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk6xj\" (UniqueName: \"kubernetes.io/projected/bc947c90-b4bc-4fc0-9935-10706af098cd-kube-api-access-hk6xj\") pod \"maas-api-revoke-sub-152b8586-d747j\" (UID: \"bc947c90-b4bc-4fc0-9935-10706af098cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"
Oct 30 12:51:49.677744 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.677628    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc947c90-b4bc-4fc0-9935-10706af098cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-152b8586-d747j\" (UID: \"bc947c90-b4bc-4fc0-9935-10706af098cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"
Oct 30 12:51:49.677817 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.677766    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc947c90-b4bc-4fc0-9935-10706af098cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-152b8586-d747j\" (UID: \"bc947c90-b4bc-4fc0-9935-10706af098cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"
Oct 30 12:51:49.678369 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.678340    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc947c90-b4bc-4fc0-9935-10706af098cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-152b8586-d747j\" (UID: \"bc947c90-b4bc-4fc0-9935-10706af098cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"
Oct 30 12:51:49.680181 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.680150    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc947c90-b4bc-4fc0-9935-10706af098cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-152b8586-d747j\" (UID: \"bc947c90-b4bc-4fc0-9935-10706af098cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"
Oct 30 12:51:49.696874 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.696850    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk6xj\" (UniqueName: \"kubernetes.io/projected/bc947c90-b4bc-4fc0-9935-10706af098cd-kube-api-access-hk6xj\") pod \"maas-api-revoke-sub-152b8586-d747j\" (UID: \"bc947c90-b4bc-4fc0-9935-10706af098cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"
Oct 30 12:51:49.790316 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:49.790293    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"
Oct 30 12:51:50.090665 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:50.090630    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"]
Oct 30 12:51:50.091211 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:51:50.091182    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc947c90_b4bc_4fc0_9935_10706af098cd.slice/crio-828e700747e938b75d54582a86c553796efef3d5f8cc49eb9be6e2353b85b7ba WatchSource:0}: Error finding container 828e700747e938b75d54582a86c553796efef3d5f8cc49eb9be6e2353b85b7ba: Status 404 returned error can't find the container with id 828e700747e938b75d54582a86c553796efef3d5f8cc49eb9be6e2353b85b7ba
Oct 30 12:51:50.893619 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:50.893588    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j" event={"ID":"bc947c90-b4bc-4fc0-9935-10706af098cd","Type":"ContainerStarted","Data":"9dd2906f0902fc6ca562e57d97838926deee5b71c8030e6d4d1d07e397fc961e"}
Oct 30 12:51:50.894022 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:50.893638    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j" event={"ID":"bc947c90-b4bc-4fc0-9935-10706af098cd","Type":"ContainerStarted","Data":"828e700747e938b75d54582a86c553796efef3d5f8cc49eb9be6e2353b85b7ba"}
Oct 30 12:51:50.931663 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:50.931571    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j" podStartSLOduration=1.931551953 podStartE2EDuration="1.931551953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:50.930301791 +0000 UTC m=+899.454586010" watchObservedRunningTime="2026-10-30 12:51:50.931551953 +0000 UTC m=+899.455836172"
Oct 30 12:51:51.758613 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:51.758588    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 12:51:51.762540 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:51.762520    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 12:51:51.937868 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:51.937831    5837 generic.go:356] "Generic (PLEG): container finished" podID="bc947c90-b4bc-4fc0-9935-10706af098cd" containerID="9dd2906f0902fc6ca562e57d97838926deee5b71c8030e6d4d1d07e397fc961e" exitCode=0
Oct 30 12:51:51.938314 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:51.937897    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j" event={"ID":"bc947c90-b4bc-4fc0-9935-10706af098cd","Type":"ContainerDied","Data":"9dd2906f0902fc6ca562e57d97838926deee5b71c8030e6d4d1d07e397fc961e"}
Oct 30 12:51:53.100829 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:53.100807    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"
Oct 30 12:51:53.125577 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:53.125557    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc947c90-b4bc-4fc0-9935-10706af098cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc947c90-b4bc-4fc0-9935-10706af098cd-maas-api-xks-ca\") pod \"bc947c90-b4bc-4fc0-9935-10706af098cd\" (UID: \"bc947c90-b4bc-4fc0-9935-10706af098cd\") "
Oct 30 12:51:53.125690 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:53.125612    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc947c90-b4bc-4fc0-9935-10706af098cd-kube-api-access-hk6xj\" (UniqueName: \"kubernetes.io/projected/bc947c90-b4bc-4fc0-9935-10706af098cd-kube-api-access-hk6xj\") pod \"bc947c90-b4bc-4fc0-9935-10706af098cd\" (UID: \"bc947c90-b4bc-4fc0-9935-10706af098cd\") "
Oct 30 12:51:53.125690 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:53.125670    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc947c90-b4bc-4fc0-9935-10706af098cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc947c90-b4bc-4fc0-9935-10706af098cd-maas-api-openshift-ca\") pod \"bc947c90-b4bc-4fc0-9935-10706af098cd\" (UID: \"bc947c90-b4bc-4fc0-9935-10706af098cd\") "
Oct 30 12:51:53.126061 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:53.126028    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc947c90-b4bc-4fc0-9935-10706af098cd-maas-api-openshift-ca" pod "bc947c90-b4bc-4fc0-9935-10706af098cd" (UID: "bc947c90-b4bc-4fc0-9935-10706af098cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:51:53.126210 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:53.126165    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc947c90-b4bc-4fc0-9935-10706af098cd-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:51:53.127796 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:53.127773    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc947c90-b4bc-4fc0-9935-10706af098cd-maas-api-xks-ca" pod "bc947c90-b4bc-4fc0-9935-10706af098cd" (UID: "bc947c90-b4bc-4fc0-9935-10706af098cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:51:53.127884 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:53.127856    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc947c90-b4bc-4fc0-9935-10706af098cd-kube-api-access-hk6xj" pod "bc947c90-b4bc-4fc0-9935-10706af098cd" (UID: "bc947c90-b4bc-4fc0-9935-10706af098cd"). InnerVolumeSpecName "kube-api-access-hk6xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:51:53.227442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:53.227358    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc947c90-b4bc-4fc0-9935-10706af098cd-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:51:53.227442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:53.227384    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk6xj\" (UniqueName: \"kubernetes.io/projected/bc947c90-b4bc-4fc0-9935-10706af098cd-kube-api-access-hk6xj\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:51:53.952362 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:53.952279    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"
Oct 30 12:51:53.952572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:53.952276    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j" event={"ID":"bc947c90-b4bc-4fc0-9935-10706af098cd","Type":"ContainerDied","Data":"828e700747e938b75d54582a86c553796efef3d5f8cc49eb9be6e2353b85b7ba"}
Oct 30 12:51:53.952572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:53.952435    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="828e700747e938b75d54582a86c553796efef3d5f8cc49eb9be6e2353b85b7ba"
Oct 30 12:51:59.811418 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:51:59.799569    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:00.400418 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:00.393538    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:52:01.408420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:01.407690    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:52:01.471417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:01.409532    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:52:01.924581 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:01.903536    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:52:01.924581 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:01.903626    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:01.990420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:01.981608    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:52:01.990420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:01.981700    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" podUID="48deaa90-e66d-454f-af97-1a5462424e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:02.372685 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:02.372648    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:52:02.377534 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:02.377496    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:02.429430 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:02.411296    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:02.840184 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:02.840121    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:52:03.483636 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:03.483575    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:52:04.440419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:04.423494    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:04.491896 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:04.491848    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:04.650419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:04.649646    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:52:13.421907 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.421877    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"]
Oct 30 12:52:13.422648 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.422628    5837 cpu_manager.go:414] "containerMap: removing container" podUID="bc947c90-b4bc-4fc0-9935-10706af098cd" containerName="revoke-keys"
Oct 30 12:52:13.422721 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.422649    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc947c90-b4bc-4fc0-9935-10706af098cd" containerName="revoke-keys"
Oct 30 12:52:13.422828 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.422816    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc947c90-b4bc-4fc0-9935-10706af098cd" containerName="revoke-keys"
Oct 30 12:52:13.426049 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.426029    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"
Oct 30 12:52:13.443613 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.443581    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"]
Oct 30 12:52:13.450767 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.450741    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:52:13.450894 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.450772    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:52:13.451928 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.451909    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:52:13.451928 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.451909    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:52:13.542515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.542489    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cbhr\" (UniqueName: \"kubernetes.io/projected/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-kube-api-access-8cbhr\") pod \"maas-api-revoke-sub-e268819f-wqxls\" (UID: \"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"
Oct 30 12:52:13.542659 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.542602    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e268819f-wqxls\" (UID: \"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"
Oct 30 12:52:13.542659 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.542624    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e268819f-wqxls\" (UID: \"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"
Oct 30 12:52:13.643590 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.643563    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cbhr\" (UniqueName: \"kubernetes.io/projected/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-kube-api-access-8cbhr\") pod \"maas-api-revoke-sub-e268819f-wqxls\" (UID: \"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"
Oct 30 12:52:13.643736 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.643653    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e268819f-wqxls\" (UID: \"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"
Oct 30 12:52:13.643736 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.643676    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e268819f-wqxls\" (UID: \"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"
Oct 30 12:52:13.644269 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.644243    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e268819f-wqxls\" (UID: \"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"
Oct 30 12:52:13.646092 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.646070    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e268819f-wqxls\" (UID: \"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"
Oct 30 12:52:13.663320 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.663290    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cbhr\" (UniqueName: \"kubernetes.io/projected/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-kube-api-access-8cbhr\") pod \"maas-api-revoke-sub-e268819f-wqxls\" (UID: \"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"
Oct 30 12:52:13.741694 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:13.741672    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"
Oct 30 12:52:14.100041 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:14.100010    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"]
Oct 30 12:52:14.100362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:52:14.100325    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6dcb13_bdc9_4635_910b_1c5bed0a3b5e.slice/crio-b748213ffbbcd821b016f2c706808fc04bd79431e34a4455ab61e99d2b9d94db WatchSource:0}: Error finding container b748213ffbbcd821b016f2c706808fc04bd79431e34a4455ab61e99d2b9d94db: Status 404 returned error can't find the container with id b748213ffbbcd821b016f2c706808fc04bd79431e34a4455ab61e99d2b9d94db
Oct 30 12:52:14.627716 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:14.627683    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls" event={"ID":"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e","Type":"ContainerStarted","Data":"732a7f64fdd70967eb28a65ce8ef47bae1e822866f1a950e0a43fffdc5a73aeb"}
Oct 30 12:52:14.628114 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:14.627768    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls" event={"ID":"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e","Type":"ContainerStarted","Data":"b748213ffbbcd821b016f2c706808fc04bd79431e34a4455ab61e99d2b9d94db"}
Oct 30 12:52:14.667588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:14.667489    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls" podStartSLOduration=1.6674746219999999 podStartE2EDuration="1.667474622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:14.664460887 +0000 UTC m=+923.188745103" watchObservedRunningTime="2026-10-30 12:52:14.667474622 +0000 UTC m=+923.191758841"
Oct 30 12:52:15.110317 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:52:15.110277    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:52:15.333631 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.333559    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"]
Oct 30 12:52:15.337923 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.337905    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"
Oct 30 12:52:15.355355 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.355329    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"]
Oct 30 12:52:15.467686 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.467652    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8478f5ef-kjw4q\" (UID: \"bef9e74c-67c5-4fb6-ad94-5e9fe269792a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"
Oct 30 12:52:15.467849 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.467777    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8478f5ef-kjw4q\" (UID: \"bef9e74c-67c5-4fb6-ad94-5e9fe269792a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"
Oct 30 12:52:15.467933 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.467917    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77tjr\" (UniqueName: \"kubernetes.io/projected/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-kube-api-access-77tjr\") pod \"maas-api-revoke-sub-8478f5ef-kjw4q\" (UID: \"bef9e74c-67c5-4fb6-ad94-5e9fe269792a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"
Oct 30 12:52:15.569405 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.569352    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8478f5ef-kjw4q\" (UID: \"bef9e74c-67c5-4fb6-ad94-5e9fe269792a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"
Oct 30 12:52:15.569584 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.569561    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8478f5ef-kjw4q\" (UID: \"bef9e74c-67c5-4fb6-ad94-5e9fe269792a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"
Oct 30 12:52:15.569841 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.569824    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77tjr\" (UniqueName: \"kubernetes.io/projected/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-kube-api-access-77tjr\") pod \"maas-api-revoke-sub-8478f5ef-kjw4q\" (UID: \"bef9e74c-67c5-4fb6-ad94-5e9fe269792a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"
Oct 30 12:52:15.570127 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.570098    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8478f5ef-kjw4q\" (UID: \"bef9e74c-67c5-4fb6-ad94-5e9fe269792a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"
Oct 30 12:52:15.572270 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.572242    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8478f5ef-kjw4q\" (UID: \"bef9e74c-67c5-4fb6-ad94-5e9fe269792a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"
Oct 30 12:52:15.589496 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.589423    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77tjr\" (UniqueName: \"kubernetes.io/projected/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-kube-api-access-77tjr\") pod \"maas-api-revoke-sub-8478f5ef-kjw4q\" (UID: \"bef9e74c-67c5-4fb6-ad94-5e9fe269792a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"
Oct 30 12:52:15.636270 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.636235    5837 generic.go:356] "Generic (PLEG): container finished" podID="fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e" containerID="732a7f64fdd70967eb28a65ce8ef47bae1e822866f1a950e0a43fffdc5a73aeb" exitCode=0
Oct 30 12:52:15.636711 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.636319    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls" event={"ID":"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e","Type":"ContainerDied","Data":"732a7f64fdd70967eb28a65ce8ef47bae1e822866f1a950e0a43fffdc5a73aeb"}
Oct 30 12:52:15.650931 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.650909    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"
Oct 30 12:52:15.952611 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:52:15.952577    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef9e74c_67c5_4fb6_ad94_5e9fe269792a.slice/crio-8d525e0c514ee0ba5959119124fb9e7b2dcfe213aa7eb37fb225a1c2bd8303a5 WatchSource:0}: Error finding container 8d525e0c514ee0ba5959119124fb9e7b2dcfe213aa7eb37fb225a1c2bd8303a5: Status 404 returned error can't find the container with id 8d525e0c514ee0ba5959119124fb9e7b2dcfe213aa7eb37fb225a1c2bd8303a5
Oct 30 12:52:15.955466 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:15.955441    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"]
Oct 30 12:52:16.646498 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:16.646215    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q" event={"ID":"bef9e74c-67c5-4fb6-ad94-5e9fe269792a","Type":"ContainerStarted","Data":"69669a4f501b994837ce86b08de9aaee0a5427a44476f3e419fa0e402710462a"}
Oct 30 12:52:16.646498 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:16.646278    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q" event={"ID":"bef9e74c-67c5-4fb6-ad94-5e9fe269792a","Type":"ContainerStarted","Data":"8d525e0c514ee0ba5959119124fb9e7b2dcfe213aa7eb37fb225a1c2bd8303a5"}
Oct 30 12:52:16.688358 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:16.688294    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q" podStartSLOduration=1.688272824 podStartE2EDuration="1.688272824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:16.686565752 +0000 UTC m=+925.210849972" watchObservedRunningTime="2026-10-30 12:52:16.688272824 +0000 UTC m=+925.212557044"
Oct 30 12:52:16.821081 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:16.821057    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"
Oct 30 12:52:16.899882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:16.899795    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-maas-api-xks-ca\") pod \"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e\" (UID: \"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e\") "
Oct 30 12:52:16.900051 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:16.899988    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-kube-api-access-8cbhr\" (UniqueName: \"kubernetes.io/projected/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-kube-api-access-8cbhr\") pod \"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e\" (UID: \"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e\") "
Oct 30 12:52:16.900121 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:16.900080    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-maas-api-openshift-ca\") pod \"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e\" (UID: \"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e\") "
Oct 30 12:52:16.900556 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:16.900522    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-maas-api-openshift-ca" pod "fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e" (UID: "fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:16.900845 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:16.900827    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:16.902773 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:16.902745    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-maas-api-xks-ca" pod "fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e" (UID: "fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:52:16.902906 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:16.902814    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-kube-api-access-8cbhr" pod "fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e" (UID: "fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e"). InnerVolumeSpecName "kube-api-access-8cbhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:52:17.002657 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:17.002631    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:17.002657 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:17.002656    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cbhr\" (UniqueName: \"kubernetes.io/projected/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e-kube-api-access-8cbhr\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:17.665240 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:17.665204    5837 generic.go:356] "Generic (PLEG): container finished" podID="bef9e74c-67c5-4fb6-ad94-5e9fe269792a" containerID="69669a4f501b994837ce86b08de9aaee0a5427a44476f3e419fa0e402710462a" exitCode=0
Oct 30 12:52:17.665709 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:17.665293    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q" event={"ID":"bef9e74c-67c5-4fb6-ad94-5e9fe269792a","Type":"ContainerDied","Data":"69669a4f501b994837ce86b08de9aaee0a5427a44476f3e419fa0e402710462a"}
Oct 30 12:52:17.666660 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:17.666643    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"
Oct 30 12:52:17.666766 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:17.666685    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls" event={"ID":"fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e","Type":"ContainerDied","Data":"b748213ffbbcd821b016f2c706808fc04bd79431e34a4455ab61e99d2b9d94db"}
Oct 30 12:52:17.666766 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:17.666723    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b748213ffbbcd821b016f2c706808fc04bd79431e34a4455ab61e99d2b9d94db"
Oct 30 12:52:18.842121 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:18.842096    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"
Oct 30 12:52:18.933835 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:18.933751    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-maas-api-openshift-ca\") pod \"bef9e74c-67c5-4fb6-ad94-5e9fe269792a\" (UID: \"bef9e74c-67c5-4fb6-ad94-5e9fe269792a\") "
Oct 30 12:52:18.933835 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:18.933817    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-maas-api-xks-ca\") pod \"bef9e74c-67c5-4fb6-ad94-5e9fe269792a\" (UID: \"bef9e74c-67c5-4fb6-ad94-5e9fe269792a\") "
Oct 30 12:52:18.934066 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:18.933861    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-kube-api-access-77tjr\" (UniqueName: \"kubernetes.io/projected/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-kube-api-access-77tjr\") pod \"bef9e74c-67c5-4fb6-ad94-5e9fe269792a\" (UID: \"bef9e74c-67c5-4fb6-ad94-5e9fe269792a\") "
Oct 30 12:52:18.934133 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:18.934087    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-maas-api-openshift-ca" pod "bef9e74c-67c5-4fb6-ad94-5e9fe269792a" (UID: "bef9e74c-67c5-4fb6-ad94-5e9fe269792a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:18.934324 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:18.934307    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:18.936029 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:18.936006    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-maas-api-xks-ca" pod "bef9e74c-67c5-4fb6-ad94-5e9fe269792a" (UID: "bef9e74c-67c5-4fb6-ad94-5e9fe269792a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:52:18.936117 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:18.936071    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-kube-api-access-77tjr" pod "bef9e74c-67c5-4fb6-ad94-5e9fe269792a" (UID: "bef9e74c-67c5-4fb6-ad94-5e9fe269792a"). InnerVolumeSpecName "kube-api-access-77tjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:52:19.036115 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:19.036080    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:19.036115 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:19.036114    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77tjr\" (UniqueName: \"kubernetes.io/projected/bef9e74c-67c5-4fb6-ad94-5e9fe269792a-kube-api-access-77tjr\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:19.680854 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:19.680830    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"
Oct 30 12:52:19.681009 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:19.680868    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q" event={"ID":"bef9e74c-67c5-4fb6-ad94-5e9fe269792a","Type":"ContainerDied","Data":"8d525e0c514ee0ba5959119124fb9e7b2dcfe213aa7eb37fb225a1c2bd8303a5"}
Oct 30 12:52:19.681009 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:19.680914    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d525e0c514ee0ba5959119124fb9e7b2dcfe213aa7eb37fb225a1c2bd8303a5"
Oct 30 12:52:25.100421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.091544    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"]
Oct 30 12:52:25.100421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.092468    5837 cpu_manager.go:414] "containerMap: removing container" podUID="bef9e74c-67c5-4fb6-ad94-5e9fe269792a" containerName="revoke-keys"
Oct 30 12:52:25.100421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.092488    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef9e74c-67c5-4fb6-ad94-5e9fe269792a" containerName="revoke-keys"
Oct 30 12:52:25.100421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.092568    5837 cpu_manager.go:414] "containerMap: removing container" podUID="fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e" containerName="revoke-keys"
Oct 30 12:52:25.100421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.092576    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e" containerName="revoke-keys"
Oct 30 12:52:25.100421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.092777    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef9e74c-67c5-4fb6-ad94-5e9fe269792a" containerName="revoke-keys"
Oct 30 12:52:25.100421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.092818    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e" containerName="revoke-keys"
Oct 30 12:52:25.105420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.101769    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:25.105420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.102188    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"]
Oct 30 12:52:25.124420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.119446    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 12:52:25.124420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.119572    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 12:52:25.132840 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.132789    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5mm8\" (UniqueName: \"kubernetes.io/projected/8f6c4e52-52fe-42c0-9930-59366f6d1881-kube-api-access-q5mm8\") pod \"payload-processing-external-model-79d5cf7c65-g7czd\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:25.133152 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.133134    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6c4e52-52fe-42c0-9930-59366f6d1881-plugins-config-volume\") pod \"payload-processing-external-model-79d5cf7c65-g7czd\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:25.133342 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.133327    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f6c4e52-52fe-42c0-9930-59366f6d1881-provider-credentials\") pod \"payload-processing-external-model-79d5cf7c65-g7czd\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:25.133485 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.133470    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f6c4e52-52fe-42c0-9930-59366f6d1881-routing-overlay-volume\") pod \"payload-processing-external-model-79d5cf7c65-g7czd\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:25.134047 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.134016    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-dddj8\""
Oct 30 12:52:25.134219 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.134207    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f9689ecfae68\""
Oct 30 12:52:25.134371 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.134360    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 12:52:25.136321 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.136265    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 12:52:25.234447 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.234381    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f6c4e52-52fe-42c0-9930-59366f6d1881-provider-credentials\") pod \"payload-processing-external-model-79d5cf7c65-g7czd\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:25.234667 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.234462    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f6c4e52-52fe-42c0-9930-59366f6d1881-routing-overlay-volume\") pod \"payload-processing-external-model-79d5cf7c65-g7czd\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:25.234667 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.234627    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5mm8\" (UniqueName: \"kubernetes.io/projected/8f6c4e52-52fe-42c0-9930-59366f6d1881-kube-api-access-q5mm8\") pod \"payload-processing-external-model-79d5cf7c65-g7czd\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:25.234794 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.234755    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6c4e52-52fe-42c0-9930-59366f6d1881-plugins-config-volume\") pod \"payload-processing-external-model-79d5cf7c65-g7czd\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:25.235198 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.235175    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f6c4e52-52fe-42c0-9930-59366f6d1881-routing-overlay-volume\") pod \"payload-processing-external-model-79d5cf7c65-g7czd\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:25.235375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.235356    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6c4e52-52fe-42c0-9930-59366f6d1881-plugins-config-volume\") pod \"payload-processing-external-model-79d5cf7c65-g7czd\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:25.237138 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.237118    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f6c4e52-52fe-42c0-9930-59366f6d1881-provider-credentials\") pod \"payload-processing-external-model-79d5cf7c65-g7czd\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:25.253855 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.253829    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5mm8\" (UniqueName: \"kubernetes.io/projected/8f6c4e52-52fe-42c0-9930-59366f6d1881-kube-api-access-q5mm8\") pod \"payload-processing-external-model-79d5cf7c65-g7czd\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:25.454828 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.453687    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:25.791991 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:25.791962    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"]
Oct 30 12:52:25.794253 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:52:25.794221    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6c4e52_52fe_42c0_9930_59366f6d1881.slice/crio-a10b7642673576082336348c6f6ee8ae358a6b426f106ab4fd0c7cdcf2cb33f6 WatchSource:0}: Error finding container a10b7642673576082336348c6f6ee8ae358a6b426f106ab4fd0c7cdcf2cb33f6: Status 404 returned error can't find the container with id a10b7642673576082336348c6f6ee8ae358a6b426f106ab4fd0c7cdcf2cb33f6
Oct 30 12:52:26.732685 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:26.732652    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd" event={"ID":"8f6c4e52-52fe-42c0-9930-59366f6d1881","Type":"ContainerStarted","Data":"e7294a8b658ce9c5006c354aa363581982a6836883422aecc29f66b44b547259"}
Oct 30 12:52:26.733171 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:26.732703    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd" event={"ID":"8f6c4e52-52fe-42c0-9930-59366f6d1881","Type":"ContainerStarted","Data":"a10b7642673576082336348c6f6ee8ae358a6b426f106ab4fd0c7cdcf2cb33f6"}
Oct 30 12:52:26.733171 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:26.732745    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:26.774375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:26.774331    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd" podStartSLOduration=1.7743189670000001 podStartE2EDuration="1.774318967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:26.772472026 +0000 UTC m=+935.296756244" watchObservedRunningTime="2026-10-30 12:52:26.774318967 +0000 UTC m=+935.298603185"
Oct 30 12:52:30.378258 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.378219    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"]
Oct 30 12:52:30.382842 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.382819    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"
Oct 30 12:52:30.400836 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.400807    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"]
Oct 30 12:52:30.407100 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.407075    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:52:30.407236 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.407157    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:52:30.407236 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.407196    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:52:30.408596 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.408576    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:52:30.589422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.586471    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347d1e37-c062-4aa0-9a35-60bc76148c83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3420d37c-hk2fs\" (UID: \"347d1e37-c062-4aa0-9a35-60bc76148c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"
Oct 30 12:52:30.589422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.586551    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347d1e37-c062-4aa0-9a35-60bc76148c83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3420d37c-hk2fs\" (UID: \"347d1e37-c062-4aa0-9a35-60bc76148c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"
Oct 30 12:52:30.589422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.586650    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h46qc\" (UniqueName: \"kubernetes.io/projected/347d1e37-c062-4aa0-9a35-60bc76148c83-kube-api-access-h46qc\") pod \"maas-api-revoke-sub-3420d37c-hk2fs\" (UID: \"347d1e37-c062-4aa0-9a35-60bc76148c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"
Oct 30 12:52:30.689476 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.688589    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347d1e37-c062-4aa0-9a35-60bc76148c83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3420d37c-hk2fs\" (UID: \"347d1e37-c062-4aa0-9a35-60bc76148c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"
Oct 30 12:52:30.689476 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.688690    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347d1e37-c062-4aa0-9a35-60bc76148c83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3420d37c-hk2fs\" (UID: \"347d1e37-c062-4aa0-9a35-60bc76148c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"
Oct 30 12:52:30.689476 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.688816    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h46qc\" (UniqueName: \"kubernetes.io/projected/347d1e37-c062-4aa0-9a35-60bc76148c83-kube-api-access-h46qc\") pod \"maas-api-revoke-sub-3420d37c-hk2fs\" (UID: \"347d1e37-c062-4aa0-9a35-60bc76148c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"
Oct 30 12:52:30.695471 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.690809    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347d1e37-c062-4aa0-9a35-60bc76148c83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3420d37c-hk2fs\" (UID: \"347d1e37-c062-4aa0-9a35-60bc76148c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"
Oct 30 12:52:30.695471 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.693109    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347d1e37-c062-4aa0-9a35-60bc76148c83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3420d37c-hk2fs\" (UID: \"347d1e37-c062-4aa0-9a35-60bc76148c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"
Oct 30 12:52:30.715775 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:30.715749    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h46qc\" (UniqueName: \"kubernetes.io/projected/347d1e37-c062-4aa0-9a35-60bc76148c83-kube-api-access-h46qc\") pod \"maas-api-revoke-sub-3420d37c-hk2fs\" (UID: \"347d1e37-c062-4aa0-9a35-60bc76148c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"
Oct 30 12:52:31.002612 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:31.002574    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"
Oct 30 12:52:32.158481 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:32.137913    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"]
Oct 30 12:52:32.566992 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:32.566946    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs" event={"ID":"347d1e37-c062-4aa0-9a35-60bc76148c83","Type":"ContainerStarted","Data":"9f992dc4804699590d952cb14d533b0182950e93625aa8a5f6748c3b84928630"}
Oct 30 12:52:33.545551 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:33.545503    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:33.700221 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:33.700158    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs" podStartSLOduration=3.7001391999999997 podStartE2EDuration="3.7001392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:33.699856594 +0000 UTC m=+942.224140814" watchObservedRunningTime="2026-10-30 12:52:33.7001392 +0000 UTC m=+942.224423420"
Oct 30 12:52:33.724078 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:33.724026    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs" event={"ID":"347d1e37-c062-4aa0-9a35-60bc76148c83","Type":"ContainerStarted","Data":"0a9525d07d347caaeef60bf1b7579dfc18f4d903a3411f638af36b0acf2add16"}
Oct 30 12:52:34.816761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:34.806053    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:34.910355 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:34.908542    5837 generic.go:356] "Generic (PLEG): container finished" podID="347d1e37-c062-4aa0-9a35-60bc76148c83" containerID="0a9525d07d347caaeef60bf1b7579dfc18f4d903a3411f638af36b0acf2add16" exitCode=0
Oct 30 12:52:34.910355 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:34.908714    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs" event={"ID":"347d1e37-c062-4aa0-9a35-60bc76148c83","Type":"ContainerDied","Data":"0a9525d07d347caaeef60bf1b7579dfc18f4d903a3411f638af36b0acf2add16"}
Oct 30 12:52:35.559425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:35.559354    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:52:35.823470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:35.813468    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:35.892423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:35.851490    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"]
Oct 30 12:52:35.988673 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:35.988640    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:52:35.989281 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:35.989264    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"
Oct 30 12:52:36.106485 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.066770    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"]
Oct 30 12:52:36.375482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.366861    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39492b3b-a3d5-4456-a570-1cd0a9677c2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0100fcb4-z59nz\" (UID: \"39492b3b-a3d5-4456-a570-1cd0a9677c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"
Oct 30 12:52:36.375785 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.375751    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39492b3b-a3d5-4456-a570-1cd0a9677c2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0100fcb4-z59nz\" (UID: \"39492b3b-a3d5-4456-a570-1cd0a9677c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"
Oct 30 12:52:36.402694 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.399543    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlf9f\" (UniqueName: \"kubernetes.io/projected/39492b3b-a3d5-4456-a570-1cd0a9677c2b-kube-api-access-vlf9f\") pod \"maas-api-revoke-sub-0100fcb4-z59nz\" (UID: \"39492b3b-a3d5-4456-a570-1cd0a9677c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"
Oct 30 12:52:36.426448 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.418806    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:52:36.426448 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.418868    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:36.512422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.503425    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39492b3b-a3d5-4456-a570-1cd0a9677c2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0100fcb4-z59nz\" (UID: \"39492b3b-a3d5-4456-a570-1cd0a9677c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"
Oct 30 12:52:36.512422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.503493    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39492b3b-a3d5-4456-a570-1cd0a9677c2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0100fcb4-z59nz\" (UID: \"39492b3b-a3d5-4456-a570-1cd0a9677c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"
Oct 30 12:52:36.512422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.503560    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlf9f\" (UniqueName: \"kubernetes.io/projected/39492b3b-a3d5-4456-a570-1cd0a9677c2b-kube-api-access-vlf9f\") pod \"maas-api-revoke-sub-0100fcb4-z59nz\" (UID: \"39492b3b-a3d5-4456-a570-1cd0a9677c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"
Oct 30 12:52:36.534695 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.534660    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39492b3b-a3d5-4456-a570-1cd0a9677c2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0100fcb4-z59nz\" (UID: \"39492b3b-a3d5-4456-a570-1cd0a9677c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"
Oct 30 12:52:36.551622 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.551586    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39492b3b-a3d5-4456-a570-1cd0a9677c2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0100fcb4-z59nz\" (UID: \"39492b3b-a3d5-4456-a570-1cd0a9677c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"
Oct 30 12:52:36.575690 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.574915    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:52:36.576256 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.576201    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:52:36.583427 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.583353    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlf9f\" (UniqueName: \"kubernetes.io/projected/39492b3b-a3d5-4456-a570-1cd0a9677c2b-kube-api-access-vlf9f\") pod \"maas-api-revoke-sub-0100fcb4-z59nz\" (UID: \"39492b3b-a3d5-4456-a570-1cd0a9677c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"
Oct 30 12:52:36.793416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.789351    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"
Oct 30 12:52:36.836153 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:36.836106    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:37.329024 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:37.328995    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:52:37.445480 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:37.428974    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:52:37.637615 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:37.637573    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:52:37.662982 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:37.662937    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:37.808684 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:37.808662    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"
Oct 30 12:52:38.321725 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:38.321691    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347d1e37-c062-4aa0-9a35-60bc76148c83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347d1e37-c062-4aa0-9a35-60bc76148c83-maas-api-openshift-ca\") pod \"347d1e37-c062-4aa0-9a35-60bc76148c83\" (UID: \"347d1e37-c062-4aa0-9a35-60bc76148c83\") "
Oct 30 12:52:38.322299 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:38.322276    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347d1e37-c062-4aa0-9a35-60bc76148c83-kube-api-access-h46qc\" (UniqueName: \"kubernetes.io/projected/347d1e37-c062-4aa0-9a35-60bc76148c83-kube-api-access-h46qc\") pod \"347d1e37-c062-4aa0-9a35-60bc76148c83\" (UID: \"347d1e37-c062-4aa0-9a35-60bc76148c83\") "
Oct 30 12:52:38.322495 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:38.322473    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/347d1e37-c062-4aa0-9a35-60bc76148c83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347d1e37-c062-4aa0-9a35-60bc76148c83-maas-api-xks-ca\") pod \"347d1e37-c062-4aa0-9a35-60bc76148c83\" (UID: \"347d1e37-c062-4aa0-9a35-60bc76148c83\") "
Oct 30 12:52:38.417524 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:38.415138    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347d1e37-c062-4aa0-9a35-60bc76148c83-maas-api-openshift-ca" pod "347d1e37-c062-4aa0-9a35-60bc76148c83" (UID: "347d1e37-c062-4aa0-9a35-60bc76148c83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:38.426258 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:38.426225    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347d1e37-c062-4aa0-9a35-60bc76148c83-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:38.504186 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:38.504157    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"
Oct 30 12:52:38.505856 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:38.505670    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347d1e37-c062-4aa0-9a35-60bc76148c83-kube-api-access-h46qc" pod "347d1e37-c062-4aa0-9a35-60bc76148c83" (UID: "347d1e37-c062-4aa0-9a35-60bc76148c83"). InnerVolumeSpecName "kube-api-access-h46qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:52:38.528479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:38.525337    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs" event={"ID":"347d1e37-c062-4aa0-9a35-60bc76148c83","Type":"ContainerDied","Data":"9f992dc4804699590d952cb14d533b0182950e93625aa8a5f6748c3b84928630"}
Oct 30 12:52:38.528479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:38.525421    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f992dc4804699590d952cb14d533b0182950e93625aa8a5f6748c3b84928630"
Oct 30 12:52:38.528479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:38.525489    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:38.541178 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:38.540056    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h46qc\" (UniqueName: \"kubernetes.io/projected/347d1e37-c062-4aa0-9a35-60bc76148c83-kube-api-access-h46qc\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:38.621897 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:38.621855    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347d1e37-c062-4aa0-9a35-60bc76148c83-maas-api-xks-ca" pod "347d1e37-c062-4aa0-9a35-60bc76148c83" (UID: "347d1e37-c062-4aa0-9a35-60bc76148c83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:52:38.649911 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:38.649102    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347d1e37-c062-4aa0-9a35-60bc76148c83-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:38.653602 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:38.653575    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"]
Oct 30 12:52:38.676738 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:52:38.676587    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39492b3b_a3d5_4456_a570_1cd0a9677c2b.slice/crio-fb61d3b5e0997aaae9e8c5ed581417415b55b1edcadde4487a8ab4f2be182ed7 WatchSource:0}: Error finding container fb61d3b5e0997aaae9e8c5ed581417415b55b1edcadde4487a8ab4f2be182ed7: Status 404 returned error can't find the container with id fb61d3b5e0997aaae9e8c5ed581417415b55b1edcadde4487a8ab4f2be182ed7
Oct 30 12:52:39.122513 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.122475    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"]
Oct 30 12:52:39.123736 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.123709    5837 cpu_manager.go:414] "containerMap: removing container" podUID="347d1e37-c062-4aa0-9a35-60bc76148c83" containerName="revoke-keys"
Oct 30 12:52:39.123867 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.123737    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347d1e37-c062-4aa0-9a35-60bc76148c83" containerName="revoke-keys"
Oct 30 12:52:39.123982 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.123968    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="347d1e37-c062-4aa0-9a35-60bc76148c83" containerName="revoke-keys"
Oct 30 12:52:39.129481 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.129305    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"
Oct 30 12:52:39.141800 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.141771    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"]
Oct 30 12:52:39.158845 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.158809    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mggxj\" (UniqueName: \"kubernetes.io/projected/c040266d-cb76-46cf-89f9-4ac58d484f14-kube-api-access-mggxj\") pod \"maas-api-revoke-sub-555fdd9c-kxwqb\" (UID: \"c040266d-cb76-46cf-89f9-4ac58d484f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"
Oct 30 12:52:39.159015 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.158960    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c040266d-cb76-46cf-89f9-4ac58d484f14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555fdd9c-kxwqb\" (UID: \"c040266d-cb76-46cf-89f9-4ac58d484f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"
Oct 30 12:52:39.159079 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.159043    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c040266d-cb76-46cf-89f9-4ac58d484f14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555fdd9c-kxwqb\" (UID: \"c040266d-cb76-46cf-89f9-4ac58d484f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"
Oct 30 12:52:39.260472 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.260441    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mggxj\" (UniqueName: \"kubernetes.io/projected/c040266d-cb76-46cf-89f9-4ac58d484f14-kube-api-access-mggxj\") pod \"maas-api-revoke-sub-555fdd9c-kxwqb\" (UID: \"c040266d-cb76-46cf-89f9-4ac58d484f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"
Oct 30 12:52:39.260668 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.260649    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c040266d-cb76-46cf-89f9-4ac58d484f14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555fdd9c-kxwqb\" (UID: \"c040266d-cb76-46cf-89f9-4ac58d484f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"
Oct 30 12:52:39.260805 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.260788    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c040266d-cb76-46cf-89f9-4ac58d484f14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555fdd9c-kxwqb\" (UID: \"c040266d-cb76-46cf-89f9-4ac58d484f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"
Oct 30 12:52:39.261297 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.261272    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c040266d-cb76-46cf-89f9-4ac58d484f14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555fdd9c-kxwqb\" (UID: \"c040266d-cb76-46cf-89f9-4ac58d484f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"
Oct 30 12:52:39.263177 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.263152    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c040266d-cb76-46cf-89f9-4ac58d484f14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555fdd9c-kxwqb\" (UID: \"c040266d-cb76-46cf-89f9-4ac58d484f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"
Oct 30 12:52:39.279891 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.279845    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mggxj\" (UniqueName: \"kubernetes.io/projected/c040266d-cb76-46cf-89f9-4ac58d484f14-kube-api-access-mggxj\") pod \"maas-api-revoke-sub-555fdd9c-kxwqb\" (UID: \"c040266d-cb76-46cf-89f9-4ac58d484f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"
Oct 30 12:52:39.452234 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.452159    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"
Oct 30 12:52:39.517435 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.516826    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz" event={"ID":"39492b3b-a3d5-4456-a570-1cd0a9677c2b","Type":"ContainerStarted","Data":"17bd30b68ad668ef4978e1cae00b2d8644ef4bd4c85957c902531f0cbf0c6d27"}
Oct 30 12:52:39.517435 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.516881    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz" event={"ID":"39492b3b-a3d5-4456-a570-1cd0a9677c2b","Type":"ContainerStarted","Data":"fb61d3b5e0997aaae9e8c5ed581417415b55b1edcadde4487a8ab4f2be182ed7"}
Oct 30 12:52:39.557963 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.557912    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz" podStartSLOduration=4.557899739 podStartE2EDuration="4.557899739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:39.553810616 +0000 UTC m=+948.078094834" watchObservedRunningTime="2026-10-30 12:52:39.557899739 +0000 UTC m=+948.082183959"
Oct 30 12:52:39.748489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:39.748460    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"]
Oct 30 12:52:39.748610 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:52:39.748520    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc040266d_cb76_46cf_89f9_4ac58d484f14.slice/crio-61e12cea89f7aab95a0cf766ab12efe778bdcbf54f743e552823fea941e51df9 WatchSource:0}: Error finding container 61e12cea89f7aab95a0cf766ab12efe778bdcbf54f743e552823fea941e51df9: Status 404 returned error can't find the container with id 61e12cea89f7aab95a0cf766ab12efe778bdcbf54f743e552823fea941e51df9
Oct 30 12:52:40.561129 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:40.561079    5837 generic.go:356] "Generic (PLEG): container finished" podID="39492b3b-a3d5-4456-a570-1cd0a9677c2b" containerID="17bd30b68ad668ef4978e1cae00b2d8644ef4bd4c85957c902531f0cbf0c6d27" exitCode=0
Oct 30 12:52:40.561696 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:40.561493    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz" event={"ID":"39492b3b-a3d5-4456-a570-1cd0a9677c2b","Type":"ContainerDied","Data":"17bd30b68ad668ef4978e1cae00b2d8644ef4bd4c85957c902531f0cbf0c6d27"}
Oct 30 12:52:40.566688 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:40.566599    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb" event={"ID":"c040266d-cb76-46cf-89f9-4ac58d484f14","Type":"ContainerStarted","Data":"067c09ce6723d1c49208435c2139dd84b1dca1776f58977d520afe59c4b3146c"}
Oct 30 12:52:40.566688 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:40.566659    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb" event={"ID":"c040266d-cb76-46cf-89f9-4ac58d484f14","Type":"ContainerStarted","Data":"61e12cea89f7aab95a0cf766ab12efe778bdcbf54f743e552823fea941e51df9"}
Oct 30 12:52:40.641661 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:40.641607    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb" podStartSLOduration=1.6415907299999999 podStartE2EDuration="1.64159073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:40.637477868 +0000 UTC m=+949.161762091" watchObservedRunningTime="2026-10-30 12:52:40.64159073 +0000 UTC m=+949.165874963"
Oct 30 12:52:41.575305 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:41.575259    5837 generic.go:356] "Generic (PLEG): container finished" podID="c040266d-cb76-46cf-89f9-4ac58d484f14" containerID="067c09ce6723d1c49208435c2139dd84b1dca1776f58977d520afe59c4b3146c" exitCode=0
Oct 30 12:52:41.575769 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:41.575335    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb" event={"ID":"c040266d-cb76-46cf-89f9-4ac58d484f14","Type":"ContainerDied","Data":"067c09ce6723d1c49208435c2139dd84b1dca1776f58977d520afe59c4b3146c"}
Oct 30 12:52:41.740709 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:41.740687    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"
Oct 30 12:52:41.801946 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:41.801916    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39492b3b-a3d5-4456-a570-1cd0a9677c2b-kube-api-access-vlf9f\" (UniqueName: \"kubernetes.io/projected/39492b3b-a3d5-4456-a570-1cd0a9677c2b-kube-api-access-vlf9f\") pod \"39492b3b-a3d5-4456-a570-1cd0a9677c2b\" (UID: \"39492b3b-a3d5-4456-a570-1cd0a9677c2b\") "
Oct 30 12:52:41.802110 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:41.802025    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39492b3b-a3d5-4456-a570-1cd0a9677c2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39492b3b-a3d5-4456-a570-1cd0a9677c2b-maas-api-xks-ca\") pod \"39492b3b-a3d5-4456-a570-1cd0a9677c2b\" (UID: \"39492b3b-a3d5-4456-a570-1cd0a9677c2b\") "
Oct 30 12:52:41.802110 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:41.802070    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39492b3b-a3d5-4456-a570-1cd0a9677c2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39492b3b-a3d5-4456-a570-1cd0a9677c2b-maas-api-openshift-ca\") pod \"39492b3b-a3d5-4456-a570-1cd0a9677c2b\" (UID: \"39492b3b-a3d5-4456-a570-1cd0a9677c2b\") "
Oct 30 12:52:41.802577 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:41.802546    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39492b3b-a3d5-4456-a570-1cd0a9677c2b-maas-api-openshift-ca" pod "39492b3b-a3d5-4456-a570-1cd0a9677c2b" (UID: "39492b3b-a3d5-4456-a570-1cd0a9677c2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:41.804366 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:41.804325    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39492b3b-a3d5-4456-a570-1cd0a9677c2b-maas-api-xks-ca" pod "39492b3b-a3d5-4456-a570-1cd0a9677c2b" (UID: "39492b3b-a3d5-4456-a570-1cd0a9677c2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:52:41.804805 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:41.804778    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39492b3b-a3d5-4456-a570-1cd0a9677c2b-kube-api-access-vlf9f" pod "39492b3b-a3d5-4456-a570-1cd0a9677c2b" (UID: "39492b3b-a3d5-4456-a570-1cd0a9677c2b"). InnerVolumeSpecName "kube-api-access-vlf9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:52:41.903425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:41.903346    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlf9f\" (UniqueName: \"kubernetes.io/projected/39492b3b-a3d5-4456-a570-1cd0a9677c2b-kube-api-access-vlf9f\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:41.903425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:41.903373    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39492b3b-a3d5-4456-a570-1cd0a9677c2b-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:41.903425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:41.903383    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39492b3b-a3d5-4456-a570-1cd0a9677c2b-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:42.583630 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:42.583594    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"
Oct 30 12:52:42.584070 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:42.583627    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz" event={"ID":"39492b3b-a3d5-4456-a570-1cd0a9677c2b","Type":"ContainerDied","Data":"fb61d3b5e0997aaae9e8c5ed581417415b55b1edcadde4487a8ab4f2be182ed7"}
Oct 30 12:52:42.584070 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:42.583678    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb61d3b5e0997aaae9e8c5ed581417415b55b1edcadde4487a8ab4f2be182ed7"
Oct 30 12:52:42.746482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:42.746458    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"
Oct 30 12:52:42.814740 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:42.814698    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c040266d-cb76-46cf-89f9-4ac58d484f14-kube-api-access-mggxj\" (UniqueName: \"kubernetes.io/projected/c040266d-cb76-46cf-89f9-4ac58d484f14-kube-api-access-mggxj\") pod \"c040266d-cb76-46cf-89f9-4ac58d484f14\" (UID: \"c040266d-cb76-46cf-89f9-4ac58d484f14\") "
Oct 30 12:52:42.814920 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:42.814753    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c040266d-cb76-46cf-89f9-4ac58d484f14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c040266d-cb76-46cf-89f9-4ac58d484f14-maas-api-openshift-ca\") pod \"c040266d-cb76-46cf-89f9-4ac58d484f14\" (UID: \"c040266d-cb76-46cf-89f9-4ac58d484f14\") "
Oct 30 12:52:42.814920 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:42.814862    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c040266d-cb76-46cf-89f9-4ac58d484f14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c040266d-cb76-46cf-89f9-4ac58d484f14-maas-api-xks-ca\") pod \"c040266d-cb76-46cf-89f9-4ac58d484f14\" (UID: \"c040266d-cb76-46cf-89f9-4ac58d484f14\") "
Oct 30 12:52:42.815262 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:42.815234    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c040266d-cb76-46cf-89f9-4ac58d484f14-maas-api-openshift-ca" pod "c040266d-cb76-46cf-89f9-4ac58d484f14" (UID: "c040266d-cb76-46cf-89f9-4ac58d484f14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:42.815608 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:42.815551    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c040266d-cb76-46cf-89f9-4ac58d484f14-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:42.817174 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:42.817152    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c040266d-cb76-46cf-89f9-4ac58d484f14-kube-api-access-mggxj" pod "c040266d-cb76-46cf-89f9-4ac58d484f14" (UID: "c040266d-cb76-46cf-89f9-4ac58d484f14"). InnerVolumeSpecName "kube-api-access-mggxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:52:42.817257 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:42.817174    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c040266d-cb76-46cf-89f9-4ac58d484f14-maas-api-xks-ca" pod "c040266d-cb76-46cf-89f9-4ac58d484f14" (UID: "c040266d-cb76-46cf-89f9-4ac58d484f14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:52:42.916933 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:42.916862    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mggxj\" (UniqueName: \"kubernetes.io/projected/c040266d-cb76-46cf-89f9-4ac58d484f14-kube-api-access-mggxj\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:42.916933 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:42.916890    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c040266d-cb76-46cf-89f9-4ac58d484f14-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:43.592149 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:43.592120    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"
Oct 30 12:52:43.592551 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:43.592151    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb" event={"ID":"c040266d-cb76-46cf-89f9-4ac58d484f14","Type":"ContainerDied","Data":"61e12cea89f7aab95a0cf766ab12efe778bdcbf54f743e552823fea941e51df9"}
Oct 30 12:52:43.592551 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:43.592196    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61e12cea89f7aab95a0cf766ab12efe778bdcbf54f743e552823fea941e51df9"
Oct 30 12:52:45.094932 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:52:45.094895    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:52:47.174847 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:47.174804    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"]
Oct 30 12:52:51.839175 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:51.839140    5837 scope.go:122] "RemoveContainer" containerID="5e6cd9925ce2c820d3d8eea86e0b5b182367b818f253fc8374f4512c81ba10b3"
Oct 30 12:52:51.859072 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:51.859024    5837 scope.go:122] "RemoveContainer" containerID="83b67b47e8a5ca7c02167f1c636faf2d905f3b6f247258aa507dee2268077310"
Oct 30 12:52:51.879536 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:51.879499    5837 scope.go:122] "RemoveContainer" containerID="da2512c7944a1f2f71a99fc107ebc59a819401f76510f3162b1e922171f1d44f"
Oct 30 12:52:51.891231 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:51.891206    5837 scope.go:122] "RemoveContainer" containerID="546ddb2149cf28e253ac4a889eab7808d0ce5a2f3ddbb11925a76aad354c8cf9"
Oct 30 12:52:51.903487 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:51.903464    5837 scope.go:122] "RemoveContainer" containerID="84ba717f92a74e60cdfde81cffc45806fc3863221dbf8a1f5cc7068ce95786e4"
Oct 30 12:52:52.175376 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.175297    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd" podUID="8f6c4e52-52fe-42c0-9930-59366f6d1881" containerName="payload-processing" containerID="cri-o://e7294a8b658ce9c5006c354aa363581982a6836883422aecc29f66b44b547259" gracePeriod=25
Oct 30 12:52:52.451009 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.450987    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:52.566002 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.565967    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f6c4e52-52fe-42c0-9930-59366f6d1881-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6c4e52-52fe-42c0-9930-59366f6d1881-plugins-config-volume\") pod \"8f6c4e52-52fe-42c0-9930-59366f6d1881\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") "
Oct 30 12:52:52.566199 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.566096    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f6c4e52-52fe-42c0-9930-59366f6d1881-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f6c4e52-52fe-42c0-9930-59366f6d1881-provider-credentials\") pod \"8f6c4e52-52fe-42c0-9930-59366f6d1881\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") "
Oct 30 12:52:52.566199 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.566185    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f6c4e52-52fe-42c0-9930-59366f6d1881-kube-api-access-q5mm8\" (UniqueName: \"kubernetes.io/projected/8f6c4e52-52fe-42c0-9930-59366f6d1881-kube-api-access-q5mm8\") pod \"8f6c4e52-52fe-42c0-9930-59366f6d1881\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") "
Oct 30 12:52:52.566336 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.566212    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f6c4e52-52fe-42c0-9930-59366f6d1881-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f6c4e52-52fe-42c0-9930-59366f6d1881-routing-overlay-volume\") pod \"8f6c4e52-52fe-42c0-9930-59366f6d1881\" (UID: \"8f6c4e52-52fe-42c0-9930-59366f6d1881\") "
Oct 30 12:52:52.566463 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.566438    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6c4e52-52fe-42c0-9930-59366f6d1881-plugins-config-volume" pod "8f6c4e52-52fe-42c0-9930-59366f6d1881" (UID: "8f6c4e52-52fe-42c0-9930-59366f6d1881"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:52.566657 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.566630    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6c4e52-52fe-42c0-9930-59366f6d1881-routing-overlay-volume" pod "8f6c4e52-52fe-42c0-9930-59366f6d1881" (UID: "8f6c4e52-52fe-42c0-9930-59366f6d1881"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:52.566949 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.566746    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6c4e52-52fe-42c0-9930-59366f6d1881-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:52.566949 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.566764    5837 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f6c4e52-52fe-42c0-9930-59366f6d1881-routing-overlay-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:52.568388 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.568362    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6c4e52-52fe-42c0-9930-59366f6d1881-provider-credentials" pod "8f6c4e52-52fe-42c0-9930-59366f6d1881" (UID: "8f6c4e52-52fe-42c0-9930-59366f6d1881"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:52:52.568490 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.568415    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6c4e52-52fe-42c0-9930-59366f6d1881-kube-api-access-q5mm8" pod "8f6c4e52-52fe-42c0-9930-59366f6d1881" (UID: "8f6c4e52-52fe-42c0-9930-59366f6d1881"). InnerVolumeSpecName "kube-api-access-q5mm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:52:52.668019 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.667987    5837 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f6c4e52-52fe-42c0-9930-59366f6d1881-provider-credentials\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:52.668019 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.668022    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5mm8\" (UniqueName: \"kubernetes.io/projected/8f6c4e52-52fe-42c0-9930-59366f6d1881-kube-api-access-q5mm8\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:52.736714 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.736685    5837 generic.go:356] "Generic (PLEG): container finished" podID="8f6c4e52-52fe-42c0-9930-59366f6d1881" containerID="e7294a8b658ce9c5006c354aa363581982a6836883422aecc29f66b44b547259" exitCode=0
Oct 30 12:52:52.736850 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.736732    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"
Oct 30 12:52:52.736850 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.736770    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd" event={"ID":"8f6c4e52-52fe-42c0-9930-59366f6d1881","Type":"ContainerDied","Data":"e7294a8b658ce9c5006c354aa363581982a6836883422aecc29f66b44b547259"}
Oct 30 12:52:52.736850 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.736829    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd" event={"ID":"8f6c4e52-52fe-42c0-9930-59366f6d1881","Type":"ContainerDied","Data":"a10b7642673576082336348c6f6ee8ae358a6b426f106ab4fd0c7cdcf2cb33f6"}
Oct 30 12:52:52.736946 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.736857    5837 scope.go:122] "RemoveContainer" containerID="e7294a8b658ce9c5006c354aa363581982a6836883422aecc29f66b44b547259"
Oct 30 12:52:52.751475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.750585    5837 scope.go:122] "RemoveContainer" containerID="e7294a8b658ce9c5006c354aa363581982a6836883422aecc29f66b44b547259"
Oct 30 12:52:52.752095 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:52:52.752062    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7294a8b658ce9c5006c354aa363581982a6836883422aecc29f66b44b547259\": container with ID starting with e7294a8b658ce9c5006c354aa363581982a6836883422aecc29f66b44b547259 not found: ID does not exist" containerID="e7294a8b658ce9c5006c354aa363581982a6836883422aecc29f66b44b547259"
Oct 30 12:52:52.752191 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.752097    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7294a8b658ce9c5006c354aa363581982a6836883422aecc29f66b44b547259"} err="failed to get container status \"e7294a8b658ce9c5006c354aa363581982a6836883422aecc29f66b44b547259\": rpc error: code = NotFound desc = could not find container \"e7294a8b658ce9c5006c354aa363581982a6836883422aecc29f66b44b547259\": container with ID starting with e7294a8b658ce9c5006c354aa363581982a6836883422aecc29f66b44b547259 not found: ID does not exist"
Oct 30 12:52:52.788019 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.787984    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"]
Oct 30 12:52:52.799202 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:52.799176    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-79d5cf7c65-g7czd"]
Oct 30 12:52:53.516639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.516608    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"]
Oct 30 12:52:53.517270 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.517254    5837 cpu_manager.go:414] "containerMap: removing container" podUID="c040266d-cb76-46cf-89f9-4ac58d484f14" containerName="revoke-keys"
Oct 30 12:52:53.517270 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.517269    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c040266d-cb76-46cf-89f9-4ac58d484f14" containerName="revoke-keys"
Oct 30 12:52:53.517489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.517281    5837 cpu_manager.go:414] "containerMap: removing container" podUID="39492b3b-a3d5-4456-a570-1cd0a9677c2b" containerName="revoke-keys"
Oct 30 12:52:53.517489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.517286    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39492b3b-a3d5-4456-a570-1cd0a9677c2b" containerName="revoke-keys"
Oct 30 12:52:53.517489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.517308    5837 cpu_manager.go:414] "containerMap: removing container" podUID="8f6c4e52-52fe-42c0-9930-59366f6d1881" containerName="payload-processing"
Oct 30 12:52:53.517489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.517313    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f6c4e52-52fe-42c0-9930-59366f6d1881" containerName="payload-processing"
Oct 30 12:52:53.517489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.517413    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="c040266d-cb76-46cf-89f9-4ac58d484f14" containerName="revoke-keys"
Oct 30 12:52:53.517489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.517423    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="39492b3b-a3d5-4456-a570-1cd0a9677c2b" containerName="revoke-keys"
Oct 30 12:52:53.517489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.517437    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f6c4e52-52fe-42c0-9930-59366f6d1881" containerName="payload-processing"
Oct 30 12:52:53.522385 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.522366    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"
Oct 30 12:52:53.538510 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.538486    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"]
Oct 30 12:52:53.546619 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.546597    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:52:53.546739 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.546662    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:52:53.546804 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.546780    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:52:53.548345 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.548328    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:52:53.611807 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.611741    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk68n\" (UniqueName: \"kubernetes.io/projected/fbdd33b4-d8b8-4273-a14d-48df02f501b8-kube-api-access-bk68n\") pod \"maas-api-revoke-sub-5c2db1d4-vfmgt\" (UID: \"fbdd33b4-d8b8-4273-a14d-48df02f501b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"
Oct 30 12:52:53.612130 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.612109    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdd33b4-d8b8-4273-a14d-48df02f501b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2db1d4-vfmgt\" (UID: \"fbdd33b4-d8b8-4273-a14d-48df02f501b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"
Oct 30 12:52:53.612687 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.612633    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdd33b4-d8b8-4273-a14d-48df02f501b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2db1d4-vfmgt\" (UID: \"fbdd33b4-d8b8-4273-a14d-48df02f501b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"
Oct 30 12:52:53.629831 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.629798    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f6c4e52-52fe-42c0-9930-59366f6d1881" path="/var/lib/kubelet/pods/8f6c4e52-52fe-42c0-9930-59366f6d1881/volumes"
Oct 30 12:52:53.714537 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.714500    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk68n\" (UniqueName: \"kubernetes.io/projected/fbdd33b4-d8b8-4273-a14d-48df02f501b8-kube-api-access-bk68n\") pod \"maas-api-revoke-sub-5c2db1d4-vfmgt\" (UID: \"fbdd33b4-d8b8-4273-a14d-48df02f501b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"
Oct 30 12:52:53.714687 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.714559    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdd33b4-d8b8-4273-a14d-48df02f501b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2db1d4-vfmgt\" (UID: \"fbdd33b4-d8b8-4273-a14d-48df02f501b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"
Oct 30 12:52:53.714760 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.714744    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdd33b4-d8b8-4273-a14d-48df02f501b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2db1d4-vfmgt\" (UID: \"fbdd33b4-d8b8-4273-a14d-48df02f501b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"
Oct 30 12:52:53.715204 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.715185    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdd33b4-d8b8-4273-a14d-48df02f501b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2db1d4-vfmgt\" (UID: \"fbdd33b4-d8b8-4273-a14d-48df02f501b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"
Oct 30 12:52:53.717249 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.717221    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdd33b4-d8b8-4273-a14d-48df02f501b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2db1d4-vfmgt\" (UID: \"fbdd33b4-d8b8-4273-a14d-48df02f501b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"
Oct 30 12:52:53.732676 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.732654    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk68n\" (UniqueName: \"kubernetes.io/projected/fbdd33b4-d8b8-4273-a14d-48df02f501b8-kube-api-access-bk68n\") pod \"maas-api-revoke-sub-5c2db1d4-vfmgt\" (UID: \"fbdd33b4-d8b8-4273-a14d-48df02f501b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"
Oct 30 12:52:53.836273 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:53.836202    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"
Oct 30 12:52:54.132949 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:54.132920    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"]
Oct 30 12:52:54.133120 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:52:54.133082    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdd33b4_d8b8_4273_a14d_48df02f501b8.slice/crio-ce87f57b3afc05c5fd91cb3ec2ce9cb4e1be33c27a64ecc9b313b6804f85989f WatchSource:0}: Error finding container ce87f57b3afc05c5fd91cb3ec2ce9cb4e1be33c27a64ecc9b313b6804f85989f: Status 404 returned error can't find the container with id ce87f57b3afc05c5fd91cb3ec2ce9cb4e1be33c27a64ecc9b313b6804f85989f
Oct 30 12:52:54.753266 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:54.753233    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt" event={"ID":"fbdd33b4-d8b8-4273-a14d-48df02f501b8","Type":"ContainerStarted","Data":"c0f70e80ee3dd36cdd79a9e8177ceb2dfbc2e57dc8ec3db1326eb4605a968596"}
Oct 30 12:52:54.753716 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:54.753283    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt" event={"ID":"fbdd33b4-d8b8-4273-a14d-48df02f501b8","Type":"ContainerStarted","Data":"ce87f57b3afc05c5fd91cb3ec2ce9cb4e1be33c27a64ecc9b313b6804f85989f"}
Oct 30 12:52:54.793587 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:54.793543    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt" podStartSLOduration=1.793529411 podStartE2EDuration="1.793529411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:54.789781345 +0000 UTC m=+963.314065567" watchObservedRunningTime="2026-10-30 12:52:54.793529411 +0000 UTC m=+963.317813629"
Oct 30 12:52:55.762310 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:55.762272    5837 generic.go:356] "Generic (PLEG): container finished" podID="fbdd33b4-d8b8-4273-a14d-48df02f501b8" containerID="c0f70e80ee3dd36cdd79a9e8177ceb2dfbc2e57dc8ec3db1326eb4605a968596" exitCode=0
Oct 30 12:52:55.762768 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:55.762334    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt" event={"ID":"fbdd33b4-d8b8-4273-a14d-48df02f501b8","Type":"ContainerDied","Data":"c0f70e80ee3dd36cdd79a9e8177ceb2dfbc2e57dc8ec3db1326eb4605a968596"}
Oct 30 12:52:56.937986 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:56.937948    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"
Oct 30 12:52:57.078740 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:57.078231    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdd33b4-d8b8-4273-a14d-48df02f501b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdd33b4-d8b8-4273-a14d-48df02f501b8-maas-api-openshift-ca\") pod \"fbdd33b4-d8b8-4273-a14d-48df02f501b8\" (UID: \"fbdd33b4-d8b8-4273-a14d-48df02f501b8\") "
Oct 30 12:52:57.078740 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:57.078310    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbdd33b4-d8b8-4273-a14d-48df02f501b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdd33b4-d8b8-4273-a14d-48df02f501b8-maas-api-xks-ca\") pod \"fbdd33b4-d8b8-4273-a14d-48df02f501b8\" (UID: \"fbdd33b4-d8b8-4273-a14d-48df02f501b8\") "
Oct 30 12:52:57.078740 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:57.078378    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdd33b4-d8b8-4273-a14d-48df02f501b8-kube-api-access-bk68n\" (UniqueName: \"kubernetes.io/projected/fbdd33b4-d8b8-4273-a14d-48df02f501b8-kube-api-access-bk68n\") pod \"fbdd33b4-d8b8-4273-a14d-48df02f501b8\" (UID: \"fbdd33b4-d8b8-4273-a14d-48df02f501b8\") "
Oct 30 12:52:57.079078 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:57.078780    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdd33b4-d8b8-4273-a14d-48df02f501b8-maas-api-openshift-ca" pod "fbdd33b4-d8b8-4273-a14d-48df02f501b8" (UID: "fbdd33b4-d8b8-4273-a14d-48df02f501b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:57.080270 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:57.079857    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdd33b4-d8b8-4273-a14d-48df02f501b8-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:57.086256 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:57.086219    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbdd33b4-d8b8-4273-a14d-48df02f501b8-maas-api-xks-ca" pod "fbdd33b4-d8b8-4273-a14d-48df02f501b8" (UID: "fbdd33b4-d8b8-4273-a14d-48df02f501b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:52:57.089110 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:57.089072    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdd33b4-d8b8-4273-a14d-48df02f501b8-kube-api-access-bk68n" pod "fbdd33b4-d8b8-4273-a14d-48df02f501b8" (UID: "fbdd33b4-d8b8-4273-a14d-48df02f501b8"). InnerVolumeSpecName "kube-api-access-bk68n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:52:57.181055 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:57.181025    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdd33b4-d8b8-4273-a14d-48df02f501b8-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:57.181055 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:57.181052    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk68n\" (UniqueName: \"kubernetes.io/projected/fbdd33b4-d8b8-4273-a14d-48df02f501b8-kube-api-access-bk68n\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:52:57.777182 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:57.777147    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"
Oct 30 12:52:57.777375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:57.777186    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt" event={"ID":"fbdd33b4-d8b8-4273-a14d-48df02f501b8","Type":"ContainerDied","Data":"ce87f57b3afc05c5fd91cb3ec2ce9cb4e1be33c27a64ecc9b313b6804f85989f"}
Oct 30 12:52:57.777375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:57.777231    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce87f57b3afc05c5fd91cb3ec2ce9cb4e1be33c27a64ecc9b313b6804f85989f"
Oct 30 12:52:58.209940 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.209905    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"]
Oct 30 12:52:58.210722 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.210701    5837 cpu_manager.go:414] "containerMap: removing container" podUID="fbdd33b4-d8b8-4273-a14d-48df02f501b8" containerName="revoke-keys"
Oct 30 12:52:58.210722 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.210723    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdd33b4-d8b8-4273-a14d-48df02f501b8" containerName="revoke-keys"
Oct 30 12:52:58.210940 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.210926    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdd33b4-d8b8-4273-a14d-48df02f501b8" containerName="revoke-keys"
Oct 30 12:52:58.214830 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.214806    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.233767 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.233741    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"]
Oct 30 12:52:58.240315 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.240286    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-53dcb1\"/\"kube-root-ca.crt\""
Oct 30 12:52:58.240585 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.240549    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-53dcb1\"/\"openshift-service-ca.crt\""
Oct 30 12:52:58.241136 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.241121    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-53dcb1\"/\"distinct-w4-53dcb1-kserve-self-signed-certs\""
Oct 30 12:52:58.241248 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.241144    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-53dcb1\"/\"default-dockercfg-v2ppp\""
Oct 30 12:52:58.294991 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.294965    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-home\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.295134 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.294999    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-tmp-dir\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.295198 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.295139    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-model-cache\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.295257 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.295207    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-dshm\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.295323 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.295303    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089cb853-6545-4a2e-9aed-3b6cede912d6-tls-certs\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.295468 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.295452    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn467\" (UniqueName: \"kubernetes.io/projected/089cb853-6545-4a2e-9aed-3b6cede912d6-kube-api-access-dn467\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.396301 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.396269    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-model-cache\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.396472 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.396320    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-dshm\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.396472 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.396378    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089cb853-6545-4a2e-9aed-3b6cede912d6-tls-certs\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.396612 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.396485    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn467\" (UniqueName: \"kubernetes.io/projected/089cb853-6545-4a2e-9aed-3b6cede912d6-kube-api-access-dn467\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.396667 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.396617    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-home\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.396667 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.396644    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-tmp-dir\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.397288 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.396826    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-model-cache\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.397288 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.397076    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-home\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.397288 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.397214    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-tmp-dir\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.416548 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.416512    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089cb853-6545-4a2e-9aed-3b6cede912d6-tls-certs\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.422647 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.422285    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-dshm\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.459416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.457033    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn467\" (UniqueName: \"kubernetes.io/projected/089cb853-6545-4a2e-9aed-3b6cede912d6-kube-api-access-dn467\") pod \"distinct-w4-53dcb1-kserve-779c4db7c-dltzc\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.533420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.532667    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:52:58.853528 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:58.853505    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"]
Oct 30 12:52:58.854011 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:52:58.853987    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod089cb853_6545_4a2e_9aed_3b6cede912d6.slice/crio-4229a79d5e8e814a1fd64d9fdcf28363421ca042e7de27d4e232fe9958e098bb WatchSource:0}: Error finding container 4229a79d5e8e814a1fd64d9fdcf28363421ca042e7de27d4e232fe9958e098bb: Status 404 returned error can't find the container with id 4229a79d5e8e814a1fd64d9fdcf28363421ca042e7de27d4e232fe9958e098bb
Oct 30 12:52:59.791845 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:59.791810    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" event={"ID":"089cb853-6545-4a2e-9aed-3b6cede912d6","Type":"ContainerStarted","Data":"4175e65a2184a1c266ff1f960db7f72fcd203e4b584fd92fc131624abe642770"}
Oct 30 12:52:59.792319 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:59.791851    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" event={"ID":"089cb853-6545-4a2e-9aed-3b6cede912d6","Type":"ContainerStarted","Data":"4229a79d5e8e814a1fd64d9fdcf28363421ca042e7de27d4e232fe9958e098bb"}
Oct 30 12:52:59.829873 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:52:59.829825    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podStartSLOduration=1.829811595 podStartE2EDuration="1.829811595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:59.828498663 +0000 UTC m=+968.352782894" watchObservedRunningTime="2026-10-30 12:52:59.829811595 +0000 UTC m=+968.354095841"
Oct 30 12:53:00.533882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:00.533853    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:53:01.909163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:01.909128    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"]
Oct 30 12:53:01.913040 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:01.913023    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"
Oct 30 12:53:01.930612 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:01.930004    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:53:01.930612 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:01.930103    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:53:01.930612 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:01.930188    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:53:01.934340 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:01.934319    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"]
Oct 30 12:53:01.939642 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:01.939621    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:53:01.946566 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:01.946539    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4033bad6-4bdkf\" (UID: \"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"
Oct 30 12:53:01.946678 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:01.946666    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qxhc\" (UniqueName: \"kubernetes.io/projected/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-kube-api-access-8qxhc\") pod \"maas-api-revoke-sub-4033bad6-4bdkf\" (UID: \"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"
Oct 30 12:53:01.946829 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:01.946815    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4033bad6-4bdkf\" (UID: \"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"
Oct 30 12:53:02.047708 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:02.047677    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qxhc\" (UniqueName: \"kubernetes.io/projected/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-kube-api-access-8qxhc\") pod \"maas-api-revoke-sub-4033bad6-4bdkf\" (UID: \"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"
Oct 30 12:53:02.047865 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:02.047779    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4033bad6-4bdkf\" (UID: \"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"
Oct 30 12:53:02.047865 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:02.047835    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4033bad6-4bdkf\" (UID: \"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"
Oct 30 12:53:02.048448 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:02.048411    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4033bad6-4bdkf\" (UID: \"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"
Oct 30 12:53:02.050335 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:02.050309    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4033bad6-4bdkf\" (UID: \"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"
Oct 30 12:53:02.066211 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:02.066185    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qxhc\" (UniqueName: \"kubernetes.io/projected/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-kube-api-access-8qxhc\") pod \"maas-api-revoke-sub-4033bad6-4bdkf\" (UID: \"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"
Oct 30 12:53:02.227068 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:02.226998    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"
Oct 30 12:53:02.540881 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:02.540827    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"]
Oct 30 12:53:02.544787 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:53:02.544752    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ca7ea2_1892_4ebc_91cb_8e1f51a1990d.slice/crio-2df767bcbee0ef1c0d16365c5998f1d7f2ead8f99388914c56926fe974b57bf3 WatchSource:0}: Error finding container 2df767bcbee0ef1c0d16365c5998f1d7f2ead8f99388914c56926fe974b57bf3: Status 404 returned error can't find the container with id 2df767bcbee0ef1c0d16365c5998f1d7f2ead8f99388914c56926fe974b57bf3
Oct 30 12:53:02.818872 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:02.818840    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf" event={"ID":"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d","Type":"ContainerStarted","Data":"2df767bcbee0ef1c0d16365c5998f1d7f2ead8f99388914c56926fe974b57bf3"}
Oct 30 12:53:03.828029 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:03.827995    5837 generic.go:356] "Generic (PLEG): container finished" podID="56ca7ea2-1892-4ebc-91cb-8e1f51a1990d" containerID="202bbed235e000aa98d097d5d918d1c4400811e2239604d541d1b4d49c974842" exitCode=0
Oct 30 12:53:03.828483 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:03.828070    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf" event={"ID":"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d","Type":"ContainerDied","Data":"202bbed235e000aa98d097d5d918d1c4400811e2239604d541d1b4d49c974842"}
Oct 30 12:53:03.931707 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:03.931667    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"]
Oct 30 12:53:03.936090 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:03.936073    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"
Oct 30 12:53:03.953861 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:03.953831    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"]
Oct 30 12:53:03.978713 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:03.978682    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x8dp\" (UniqueName: \"kubernetes.io/projected/4224b1c1-3469-4374-be0c-dcfa6036aa01-kube-api-access-7x8dp\") pod \"maas-api-revoke-sub-a32dd90d-d5jvh\" (UID: \"4224b1c1-3469-4374-be0c-dcfa6036aa01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"
Oct 30 12:53:03.978865 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:03.978752    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4224b1c1-3469-4374-be0c-dcfa6036aa01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a32dd90d-d5jvh\" (UID: \"4224b1c1-3469-4374-be0c-dcfa6036aa01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"
Oct 30 12:53:03.978865 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:03.978805    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4224b1c1-3469-4374-be0c-dcfa6036aa01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a32dd90d-d5jvh\" (UID: \"4224b1c1-3469-4374-be0c-dcfa6036aa01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"
Oct 30 12:53:04.080264 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:04.080185    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4224b1c1-3469-4374-be0c-dcfa6036aa01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a32dd90d-d5jvh\" (UID: \"4224b1c1-3469-4374-be0c-dcfa6036aa01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"
Oct 30 12:53:04.080264 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:04.080219    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4224b1c1-3469-4374-be0c-dcfa6036aa01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a32dd90d-d5jvh\" (UID: \"4224b1c1-3469-4374-be0c-dcfa6036aa01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"
Oct 30 12:53:04.080582 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:04.080453    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x8dp\" (UniqueName: \"kubernetes.io/projected/4224b1c1-3469-4374-be0c-dcfa6036aa01-kube-api-access-7x8dp\") pod \"maas-api-revoke-sub-a32dd90d-d5jvh\" (UID: \"4224b1c1-3469-4374-be0c-dcfa6036aa01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"
Oct 30 12:53:04.080988 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:04.080963    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4224b1c1-3469-4374-be0c-dcfa6036aa01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a32dd90d-d5jvh\" (UID: \"4224b1c1-3469-4374-be0c-dcfa6036aa01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"
Oct 30 12:53:04.082726 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:04.082707    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4224b1c1-3469-4374-be0c-dcfa6036aa01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a32dd90d-d5jvh\" (UID: \"4224b1c1-3469-4374-be0c-dcfa6036aa01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"
Oct 30 12:53:04.102183 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:04.102159    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x8dp\" (UniqueName: \"kubernetes.io/projected/4224b1c1-3469-4374-be0c-dcfa6036aa01-kube-api-access-7x8dp\") pod \"maas-api-revoke-sub-a32dd90d-d5jvh\" (UID: \"4224b1c1-3469-4374-be0c-dcfa6036aa01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"
Oct 30 12:53:04.250036 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:04.250005    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"
Oct 30 12:53:04.547457 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:04.547427    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"]
Oct 30 12:53:04.548089 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:53:04.548065    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4224b1c1_3469_4374_be0c_dcfa6036aa01.slice/crio-eab613e6a49245c5a9e05805adc5653be9149db4f8f2f53009fd3064c416585e WatchSource:0}: Error finding container eab613e6a49245c5a9e05805adc5653be9149db4f8f2f53009fd3064c416585e: Status 404 returned error can't find the container with id eab613e6a49245c5a9e05805adc5653be9149db4f8f2f53009fd3064c416585e
Oct 30 12:53:04.836853 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:04.836823    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh" event={"ID":"4224b1c1-3469-4374-be0c-dcfa6036aa01","Type":"ContainerStarted","Data":"750487f73b3567f9a1cca9c6b8899719c37596c8f3ff6e80ff46fe85d608d404"}
Oct 30 12:53:04.837199 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:04.836880    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh" event={"ID":"4224b1c1-3469-4374-be0c-dcfa6036aa01","Type":"ContainerStarted","Data":"eab613e6a49245c5a9e05805adc5653be9149db4f8f2f53009fd3064c416585e"}
Oct 30 12:53:04.877181 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:04.877133    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh" podStartSLOduration=1.877117546 podStartE2EDuration="1.877117546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:04.873205259 +0000 UTC m=+973.397489478" watchObservedRunningTime="2026-10-30 12:53:04.877117546 +0000 UTC m=+973.401401765"
Oct 30 12:53:04.989256 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:04.989235    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"
Oct 30 12:53:05.092050 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:05.091981    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-maas-api-xks-ca\") pod \"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d\" (UID: \"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d\") "
Oct 30 12:53:05.092203 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:05.092052    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-maas-api-openshift-ca\") pod \"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d\" (UID: \"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d\") "
Oct 30 12:53:05.092203 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:05.092104    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-kube-api-access-8qxhc\" (UniqueName: \"kubernetes.io/projected/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-kube-api-access-8qxhc\") pod \"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d\" (UID: \"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d\") "
Oct 30 12:53:05.092455 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:05.092425    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-maas-api-openshift-ca" pod "56ca7ea2-1892-4ebc-91cb-8e1f51a1990d" (UID: "56ca7ea2-1892-4ebc-91cb-8e1f51a1990d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:05.092590 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:05.092551    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:05.094360 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:05.094333    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-kube-api-access-8qxhc" pod "56ca7ea2-1892-4ebc-91cb-8e1f51a1990d" (UID: "56ca7ea2-1892-4ebc-91cb-8e1f51a1990d"). InnerVolumeSpecName "kube-api-access-8qxhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:05.094492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:05.094337    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-maas-api-xks-ca" pod "56ca7ea2-1892-4ebc-91cb-8e1f51a1990d" (UID: "56ca7ea2-1892-4ebc-91cb-8e1f51a1990d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:05.194099 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:05.194070    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:05.194099 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:05.194095    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qxhc\" (UniqueName: \"kubernetes.io/projected/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d-kube-api-access-8qxhc\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:05.845127 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:05.845096    5837 generic.go:356] "Generic (PLEG): container finished" podID="4224b1c1-3469-4374-be0c-dcfa6036aa01" containerID="750487f73b3567f9a1cca9c6b8899719c37596c8f3ff6e80ff46fe85d608d404" exitCode=0
Oct 30 12:53:05.845559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:05.845174    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh" event={"ID":"4224b1c1-3469-4374-be0c-dcfa6036aa01","Type":"ContainerDied","Data":"750487f73b3567f9a1cca9c6b8899719c37596c8f3ff6e80ff46fe85d608d404"}
Oct 30 12:53:05.846651 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:05.846625    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf" event={"ID":"56ca7ea2-1892-4ebc-91cb-8e1f51a1990d","Type":"ContainerDied","Data":"2df767bcbee0ef1c0d16365c5998f1d7f2ead8f99388914c56926fe974b57bf3"}
Oct 30 12:53:05.846651 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:05.846649    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"
Oct 30 12:53:05.846812 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:05.846674    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2df767bcbee0ef1c0d16365c5998f1d7f2ead8f99388914c56926fe974b57bf3"
Oct 30 12:53:07.035334 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:07.035309    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"
Oct 30 12:53:07.124634 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:07.124549    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4224b1c1-3469-4374-be0c-dcfa6036aa01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4224b1c1-3469-4374-be0c-dcfa6036aa01-maas-api-xks-ca\") pod \"4224b1c1-3469-4374-be0c-dcfa6036aa01\" (UID: \"4224b1c1-3469-4374-be0c-dcfa6036aa01\") "
Oct 30 12:53:07.124789 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:07.124633    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4224b1c1-3469-4374-be0c-dcfa6036aa01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4224b1c1-3469-4374-be0c-dcfa6036aa01-maas-api-openshift-ca\") pod \"4224b1c1-3469-4374-be0c-dcfa6036aa01\" (UID: \"4224b1c1-3469-4374-be0c-dcfa6036aa01\") "
Oct 30 12:53:07.124789 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:07.124704    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4224b1c1-3469-4374-be0c-dcfa6036aa01-kube-api-access-7x8dp\" (UniqueName: \"kubernetes.io/projected/4224b1c1-3469-4374-be0c-dcfa6036aa01-kube-api-access-7x8dp\") pod \"4224b1c1-3469-4374-be0c-dcfa6036aa01\" (UID: \"4224b1c1-3469-4374-be0c-dcfa6036aa01\") "
Oct 30 12:53:07.125096 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:07.125065    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4224b1c1-3469-4374-be0c-dcfa6036aa01-maas-api-openshift-ca" pod "4224b1c1-3469-4374-be0c-dcfa6036aa01" (UID: "4224b1c1-3469-4374-be0c-dcfa6036aa01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:07.125536 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:07.125512    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4224b1c1-3469-4374-be0c-dcfa6036aa01-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:07.127492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:07.127470    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4224b1c1-3469-4374-be0c-dcfa6036aa01-maas-api-xks-ca" pod "4224b1c1-3469-4374-be0c-dcfa6036aa01" (UID: "4224b1c1-3469-4374-be0c-dcfa6036aa01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:07.127600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:07.127493    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4224b1c1-3469-4374-be0c-dcfa6036aa01-kube-api-access-7x8dp" pod "4224b1c1-3469-4374-be0c-dcfa6036aa01" (UID: "4224b1c1-3469-4374-be0c-dcfa6036aa01"). InnerVolumeSpecName "kube-api-access-7x8dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:07.226327 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:07.226298    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x8dp\" (UniqueName: \"kubernetes.io/projected/4224b1c1-3469-4374-be0c-dcfa6036aa01-kube-api-access-7x8dp\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:07.226327 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:07.226325    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4224b1c1-3469-4374-be0c-dcfa6036aa01-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:07.862230 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:07.862205    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"
Oct 30 12:53:07.862439 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:07.862236    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh" event={"ID":"4224b1c1-3469-4374-be0c-dcfa6036aa01","Type":"ContainerDied","Data":"eab613e6a49245c5a9e05805adc5653be9149db4f8f2f53009fd3064c416585e"}
Oct 30 12:53:07.862439 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:07.862280    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eab613e6a49245c5a9e05805adc5653be9149db4f8f2f53009fd3064c416585e"
Oct 30 12:53:08.534599 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:08.534572    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:53:08.547080 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:08.547051    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:53:08.881173 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:08.881096    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:53:12.253808 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.253769    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"]
Oct 30 12:53:12.254423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.254408    5837 cpu_manager.go:414] "containerMap: removing container" podUID="4224b1c1-3469-4374-be0c-dcfa6036aa01" containerName="revoke-keys"
Oct 30 12:53:12.254475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.254424    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4224b1c1-3469-4374-be0c-dcfa6036aa01" containerName="revoke-keys"
Oct 30 12:53:12.254475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.254450    5837 cpu_manager.go:414] "containerMap: removing container" podUID="56ca7ea2-1892-4ebc-91cb-8e1f51a1990d" containerName="revoke-keys"
Oct 30 12:53:12.254475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.254455    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ca7ea2-1892-4ebc-91cb-8e1f51a1990d" containerName="revoke-keys"
Oct 30 12:53:12.254571 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.254565    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4224b1c1-3469-4374-be0c-dcfa6036aa01" containerName="revoke-keys"
Oct 30 12:53:12.254615 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.254579    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ca7ea2-1892-4ebc-91cb-8e1f51a1990d" containerName="revoke-keys"
Oct 30 12:53:12.258523 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.258505    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"
Oct 30 12:53:12.277120 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.277090    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"]
Oct 30 12:53:12.283823 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.283787    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:53:12.283966 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.283886    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:53:12.283966 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.283962    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:53:12.284074 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.284008    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:53:12.295922 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.295893    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bd0e26e-9dl2b\" (UID: \"3a803fc4-02c2-45f6-9c33-9153bd8f7c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"
Oct 30 12:53:12.296063 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.296033    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bd0e26e-9dl2b\" (UID: \"3a803fc4-02c2-45f6-9c33-9153bd8f7c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"
Oct 30 12:53:12.296299 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.296274    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r64q\" (UniqueName: \"kubernetes.io/projected/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-kube-api-access-2r64q\") pod \"maas-api-revoke-sub-3bd0e26e-9dl2b\" (UID: \"3a803fc4-02c2-45f6-9c33-9153bd8f7c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"
Oct 30 12:53:12.397957 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.397919    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r64q\" (UniqueName: \"kubernetes.io/projected/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-kube-api-access-2r64q\") pod \"maas-api-revoke-sub-3bd0e26e-9dl2b\" (UID: \"3a803fc4-02c2-45f6-9c33-9153bd8f7c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"
Oct 30 12:53:12.398128 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.398093    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bd0e26e-9dl2b\" (UID: \"3a803fc4-02c2-45f6-9c33-9153bd8f7c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"
Oct 30 12:53:12.398208 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.398161    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bd0e26e-9dl2b\" (UID: \"3a803fc4-02c2-45f6-9c33-9153bd8f7c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"
Oct 30 12:53:12.398981 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.398958    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bd0e26e-9dl2b\" (UID: \"3a803fc4-02c2-45f6-9c33-9153bd8f7c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"
Oct 30 12:53:12.401232 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.401166    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bd0e26e-9dl2b\" (UID: \"3a803fc4-02c2-45f6-9c33-9153bd8f7c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"
Oct 30 12:53:12.422902 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.422876    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r64q\" (UniqueName: \"kubernetes.io/projected/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-kube-api-access-2r64q\") pod \"maas-api-revoke-sub-3bd0e26e-9dl2b\" (UID: \"3a803fc4-02c2-45f6-9c33-9153bd8f7c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"
Oct 30 12:53:12.575559 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:12.575466    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"
Oct 30 12:53:13.002724 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:13.002687    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"]
Oct 30 12:53:13.044903 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:53:13.040299    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a803fc4_02c2_45f6_9c33_9153bd8f7c06.slice/crio-f8909550df8224cb9b1d854adf176c44cd7c24a10c15dc695cf5f1e1dcc22a96 WatchSource:0}: Error finding container f8909550df8224cb9b1d854adf176c44cd7c24a10c15dc695cf5f1e1dcc22a96: Status 404 returned error can't find the container with id f8909550df8224cb9b1d854adf176c44cd7c24a10c15dc695cf5f1e1dcc22a96
Oct 30 12:53:14.168106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:14.168041    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b" event={"ID":"3a803fc4-02c2-45f6-9c33-9153bd8f7c06","Type":"ContainerStarted","Data":"89323964a29e56d585a3928ec4c2921558117221e541a0f10ea62c445b50f81f"}
Oct 30 12:53:14.168682 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:14.168176    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b" event={"ID":"3a803fc4-02c2-45f6-9c33-9153bd8f7c06","Type":"ContainerStarted","Data":"f8909550df8224cb9b1d854adf176c44cd7c24a10c15dc695cf5f1e1dcc22a96"}
Oct 30 12:53:14.220417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:14.209572    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b" podStartSLOduration=2.209553634 podStartE2EDuration="2.209553634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:14.208556802 +0000 UTC m=+982.732841072" watchObservedRunningTime="2026-10-30 12:53:14.209553634 +0000 UTC m=+982.733837856"
Oct 30 12:53:14.876966 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:14.876510    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:15.200072 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:15.199978    5837 generic.go:356] "Generic (PLEG): container finished" podID="3a803fc4-02c2-45f6-9c33-9153bd8f7c06" containerID="89323964a29e56d585a3928ec4c2921558117221e541a0f10ea62c445b50f81f" exitCode=0
Oct 30 12:53:15.200791 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:15.200767    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b" event={"ID":"3a803fc4-02c2-45f6-9c33-9153bd8f7c06","Type":"ContainerDied","Data":"89323964a29e56d585a3928ec4c2921558117221e541a0f10ea62c445b50f81f"}
Oct 30 12:53:15.364509 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:15.364461    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:53:15.365145 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:53:15.365119    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:53:15.447892 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:15.447847    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:15.983456 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:15.947828    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:53:16.122644 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:16.122415    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:53:16.368760 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:16.368666    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:16.382026 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:16.382000    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:53:16.956422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:16.949087    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:53:17.381427 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:17.371617    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:17.576638 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:17.576573    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:53:17.641851 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:17.641748    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms" podUID="b35e847c-8794-4355-8d45-f474df36feee" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:17.912521 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:17.868628    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"]
Oct 30 12:53:17.941773 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:17.941742    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"
Oct 30 12:53:17.961922 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:17.961883    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:18.045511 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:18.045468    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"]
Oct 30 12:53:18.070638 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:18.070593    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rc7l\" (UniqueName: \"kubernetes.io/projected/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-kube-api-access-7rc7l\") pod \"maas-api-revoke-sub-287e87c2-rn5l4\" (UID: \"c1755206-b5ef-4c8e-aeaa-d90b5317e36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"
Oct 30 12:53:18.070970 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:18.070946    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-287e87c2-rn5l4\" (UID: \"c1755206-b5ef-4c8e-aeaa-d90b5317e36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"
Oct 30 12:53:18.071367 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:18.071349    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-287e87c2-rn5l4\" (UID: \"c1755206-b5ef-4c8e-aeaa-d90b5317e36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"
Oct 30 12:53:18.238754 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:18.226107    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-287e87c2-rn5l4\" (UID: \"c1755206-b5ef-4c8e-aeaa-d90b5317e36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"
Oct 30 12:53:18.260217 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:18.250588    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rc7l\" (UniqueName: \"kubernetes.io/projected/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-kube-api-access-7rc7l\") pod \"maas-api-revoke-sub-287e87c2-rn5l4\" (UID: \"c1755206-b5ef-4c8e-aeaa-d90b5317e36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"
Oct 30 12:53:18.303613 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:18.303559    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-287e87c2-rn5l4\" (UID: \"c1755206-b5ef-4c8e-aeaa-d90b5317e36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"
Oct 30 12:53:18.371160 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:18.371114    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-287e87c2-rn5l4\" (UID: \"c1755206-b5ef-4c8e-aeaa-d90b5317e36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"
Oct 30 12:53:18.374699 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:18.374634    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:18.458479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:18.455223    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rc7l\" (UniqueName: \"kubernetes.io/projected/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-kube-api-access-7rc7l\") pod \"maas-api-revoke-sub-287e87c2-rn5l4\" (UID: \"c1755206-b5ef-4c8e-aeaa-d90b5317e36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"
Oct 30 12:53:18.465423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:18.459825    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-287e87c2-rn5l4\" (UID: \"c1755206-b5ef-4c8e-aeaa-d90b5317e36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"
Oct 30 12:53:18.634420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:18.616357    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:53:18.661419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:18.635187    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:53:18.806603 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:18.781178    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"
Oct 30 12:53:19.011420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:18.995676    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:19.260520 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:19.260488    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"
Oct 30 12:53:19.326109 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:19.326075    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-maas-api-xks-ca\") pod \"3a803fc4-02c2-45f6-9c33-9153bd8f7c06\" (UID: \"3a803fc4-02c2-45f6-9c33-9153bd8f7c06\") "
Oct 30 12:53:19.352793 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:19.352757    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-maas-api-xks-ca" pod "3a803fc4-02c2-45f6-9c33-9153bd8f7c06" (UID: "3a803fc4-02c2-45f6-9c33-9153bd8f7c06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:19.452422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:19.427916    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-kube-api-access-2r64q\" (UniqueName: \"kubernetes.io/projected/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-kube-api-access-2r64q\") pod \"3a803fc4-02c2-45f6-9c33-9153bd8f7c06\" (UID: \"3a803fc4-02c2-45f6-9c33-9153bd8f7c06\") "
Oct 30 12:53:19.461419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:19.457564    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-maas-api-openshift-ca\") pod \"3a803fc4-02c2-45f6-9c33-9153bd8f7c06\" (UID: \"3a803fc4-02c2-45f6-9c33-9153bd8f7c06\") "
Oct 30 12:53:19.524469 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:19.487365    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:19.532420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:19.521617    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-maas-api-openshift-ca" pod "3a803fc4-02c2-45f6-9c33-9153bd8f7c06" (UID: "3a803fc4-02c2-45f6-9c33-9153bd8f7c06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:19.624522 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:19.624478    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:53:19.640422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:19.624498    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:53:19.649578 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:19.632065    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:19.658419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:19.656860    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": context deadline exceeded"
Oct 30 12:53:19.742227 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:19.741581    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-kube-api-access-2r64q" pod "3a803fc4-02c2-45f6-9c33-9153bd8f7c06" (UID: "3a803fc4-02c2-45f6-9c33-9153bd8f7c06"). InnerVolumeSpecName "kube-api-access-2r64q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:19.755695 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:19.755649    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r64q\" (UniqueName: \"kubernetes.io/projected/3a803fc4-02c2-45f6-9c33-9153bd8f7c06-kube-api-access-2r64q\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:20.103421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:20.102432    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:20.288023 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:20.287842    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b" event={"ID":"3a803fc4-02c2-45f6-9c33-9153bd8f7c06","Type":"ContainerDied","Data":"f8909550df8224cb9b1d854adf176c44cd7c24a10c15dc695cf5f1e1dcc22a96"}
Oct 30 12:53:20.288023 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:20.287903    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8909550df8224cb9b1d854adf176c44cd7c24a10c15dc695cf5f1e1dcc22a96"
Oct 30 12:53:20.288023 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:20.287980    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"
Oct 30 12:53:20.307736 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:20.307705    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"]
Oct 30 12:53:20.368855 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:53:20.368818    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1755206_b5ef_4c8e_aeaa_d90b5317e36e.slice/crio-01751f027fc34578243e95bbffb4153fde26aa0dbaa3e4a50ed2f0e1d804a865 WatchSource:0}: Error finding container 01751f027fc34578243e95bbffb4153fde26aa0dbaa3e4a50ed2f0e1d804a865: Status 404 returned error can't find the container with id 01751f027fc34578243e95bbffb4153fde26aa0dbaa3e4a50ed2f0e1d804a865
Oct 30 12:53:20.539836 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:20.539755    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:53:20.594420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:20.593133    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:53:20.706542 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:20.699502    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded"
Oct 30 12:53:21.104373 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:21.103641    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:21.417473 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:21.416793    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4" event={"ID":"c1755206-b5ef-4c8e-aeaa-d90b5317e36e","Type":"ContainerStarted","Data":"01751f027fc34578243e95bbffb4153fde26aa0dbaa3e4a50ed2f0e1d804a865"}
Oct 30 12:53:21.463512 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:21.454368    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms" podUID="b35e847c-8794-4355-8d45-f474df36feee" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:21.648419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:21.644388    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:53:22.427051 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:22.426972    5837 generic.go:356] "Generic (PLEG): container finished" podID="c1755206-b5ef-4c8e-aeaa-d90b5317e36e" containerID="abf4939e5998dbe659db1606f09c5d4a1df4ebc41991bb972fd73abbd0f766a6" exitCode=0
Oct 30 12:53:22.427051 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:22.427022    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4" event={"ID":"c1755206-b5ef-4c8e-aeaa-d90b5317e36e","Type":"ContainerDied","Data":"abf4939e5998dbe659db1606f09c5d4a1df4ebc41991bb972fd73abbd0f766a6"}
Oct 30 12:53:22.845472 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:22.845438    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"]
Oct 30 12:53:22.846297 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:22.846276    5837 cpu_manager.go:414] "containerMap: removing container" podUID="3a803fc4-02c2-45f6-9c33-9153bd8f7c06" containerName="revoke-keys"
Oct 30 12:53:22.846297 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:22.846297    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a803fc4-02c2-45f6-9c33-9153bd8f7c06" containerName="revoke-keys"
Oct 30 12:53:22.846585 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:22.846569    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a803fc4-02c2-45f6-9c33-9153bd8f7c06" containerName="revoke-keys"
Oct 30 12:53:22.850224 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:22.850205    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"
Oct 30 12:53:22.867353 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:22.867331    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"]
Oct 30 12:53:22.976500 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:22.976464    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d786e2d8-d256-456c-8885-56f246e71949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d491f279-9crps\" (UID: \"d786e2d8-d256-456c-8885-56f246e71949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"
Oct 30 12:53:22.976632 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:22.976538    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d786e2d8-d256-456c-8885-56f246e71949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d491f279-9crps\" (UID: \"d786e2d8-d256-456c-8885-56f246e71949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"
Oct 30 12:53:22.976714 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:22.976701    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s4mq\" (UniqueName: \"kubernetes.io/projected/d786e2d8-d256-456c-8885-56f246e71949-kube-api-access-8s4mq\") pod \"maas-api-revoke-sub-d491f279-9crps\" (UID: \"d786e2d8-d256-456c-8885-56f246e71949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"
Oct 30 12:53:23.077539 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.077502    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s4mq\" (UniqueName: \"kubernetes.io/projected/d786e2d8-d256-456c-8885-56f246e71949-kube-api-access-8s4mq\") pod \"maas-api-revoke-sub-d491f279-9crps\" (UID: \"d786e2d8-d256-456c-8885-56f246e71949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"
Oct 30 12:53:23.077706 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.077648    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d786e2d8-d256-456c-8885-56f246e71949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d491f279-9crps\" (UID: \"d786e2d8-d256-456c-8885-56f246e71949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"
Oct 30 12:53:23.077706 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.077678    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d786e2d8-d256-456c-8885-56f246e71949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d491f279-9crps\" (UID: \"d786e2d8-d256-456c-8885-56f246e71949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"
Oct 30 12:53:23.078373 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.078354    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d786e2d8-d256-456c-8885-56f246e71949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d491f279-9crps\" (UID: \"d786e2d8-d256-456c-8885-56f246e71949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"
Oct 30 12:53:23.080207 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.080190    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d786e2d8-d256-456c-8885-56f246e71949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d491f279-9crps\" (UID: \"d786e2d8-d256-456c-8885-56f246e71949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"
Oct 30 12:53:23.095621 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.095567    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s4mq\" (UniqueName: \"kubernetes.io/projected/d786e2d8-d256-456c-8885-56f246e71949-kube-api-access-8s4mq\") pod \"maas-api-revoke-sub-d491f279-9crps\" (UID: \"d786e2d8-d256-456c-8885-56f246e71949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"
Oct 30 12:53:23.163661 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.163636    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"
Oct 30 12:53:23.464820 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.464791    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"]
Oct 30 12:53:23.465003 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:53:23.464976    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd786e2d8_d256_456c_8885_56f246e71949.slice/crio-92501b6b9c51eac32393002ce28472853747cd4f99833a97df18134ab9181f7b WatchSource:0}: Error finding container 92501b6b9c51eac32393002ce28472853747cd4f99833a97df18134ab9181f7b: Status 404 returned error can't find the container with id 92501b6b9c51eac32393002ce28472853747cd4f99833a97df18134ab9181f7b
Oct 30 12:53:23.579781 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.579755    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"
Oct 30 12:53:23.685025 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.684955    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-kube-api-access-7rc7l\" (UniqueName: \"kubernetes.io/projected/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-kube-api-access-7rc7l\") pod \"c1755206-b5ef-4c8e-aeaa-d90b5317e36e\" (UID: \"c1755206-b5ef-4c8e-aeaa-d90b5317e36e\") "
Oct 30 12:53:23.685160 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.685064    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-maas-api-xks-ca\") pod \"c1755206-b5ef-4c8e-aeaa-d90b5317e36e\" (UID: \"c1755206-b5ef-4c8e-aeaa-d90b5317e36e\") "
Oct 30 12:53:23.685220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.685171    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-maas-api-openshift-ca\") pod \"c1755206-b5ef-4c8e-aeaa-d90b5317e36e\" (UID: \"c1755206-b5ef-4c8e-aeaa-d90b5317e36e\") "
Oct 30 12:53:23.685572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.685543    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-maas-api-openshift-ca" pod "c1755206-b5ef-4c8e-aeaa-d90b5317e36e" (UID: "c1755206-b5ef-4c8e-aeaa-d90b5317e36e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:23.685749 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.685734    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:23.687340 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.687313    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-maas-api-xks-ca" pod "c1755206-b5ef-4c8e-aeaa-d90b5317e36e" (UID: "c1755206-b5ef-4c8e-aeaa-d90b5317e36e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:23.687465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.687363    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-kube-api-access-7rc7l" pod "c1755206-b5ef-4c8e-aeaa-d90b5317e36e" (UID: "c1755206-b5ef-4c8e-aeaa-d90b5317e36e"). InnerVolumeSpecName "kube-api-access-7rc7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:23.787085 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.787058    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rc7l\" (UniqueName: \"kubernetes.io/projected/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-kube-api-access-7rc7l\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:23.787085 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:23.787085    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1755206-b5ef-4c8e-aeaa-d90b5317e36e-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:24.442847 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:24.442807    5837 generic.go:356] "Generic (PLEG): container finished" podID="d786e2d8-d256-456c-8885-56f246e71949" containerID="9e46a059e169bc4f7501fba51284d17fc91342d4dcb84508ed36bb9c4ad416f6" exitCode=0
Oct 30 12:53:24.443206 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:24.442881    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps" event={"ID":"d786e2d8-d256-456c-8885-56f246e71949","Type":"ContainerDied","Data":"9e46a059e169bc4f7501fba51284d17fc91342d4dcb84508ed36bb9c4ad416f6"}
Oct 30 12:53:24.443206 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:24.442932    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps" event={"ID":"d786e2d8-d256-456c-8885-56f246e71949","Type":"ContainerStarted","Data":"92501b6b9c51eac32393002ce28472853747cd4f99833a97df18134ab9181f7b"}
Oct 30 12:53:24.444146 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:24.444132    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"
Oct 30 12:53:24.444228 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:24.444173    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4" event={"ID":"c1755206-b5ef-4c8e-aeaa-d90b5317e36e","Type":"ContainerDied","Data":"01751f027fc34578243e95bbffb4153fde26aa0dbaa3e4a50ed2f0e1d804a865"}
Oct 30 12:53:24.444228 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:24.444215    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01751f027fc34578243e95bbffb4153fde26aa0dbaa3e4a50ed2f0e1d804a865"
Oct 30 12:53:25.653434 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:25.653067    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"
Oct 30 12:53:25.715600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:25.715529    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d786e2d8-d256-456c-8885-56f246e71949-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d786e2d8-d256-456c-8885-56f246e71949-maas-api-openshift-ca\") pod \"d786e2d8-d256-456c-8885-56f246e71949\" (UID: \"d786e2d8-d256-456c-8885-56f246e71949\") "
Oct 30 12:53:25.715730 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:25.715597    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d786e2d8-d256-456c-8885-56f246e71949-kube-api-access-8s4mq\" (UniqueName: \"kubernetes.io/projected/d786e2d8-d256-456c-8885-56f246e71949-kube-api-access-8s4mq\") pod \"d786e2d8-d256-456c-8885-56f246e71949\" (UID: \"d786e2d8-d256-456c-8885-56f246e71949\") "
Oct 30 12:53:25.715730 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:25.715685    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d786e2d8-d256-456c-8885-56f246e71949-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d786e2d8-d256-456c-8885-56f246e71949-maas-api-xks-ca\") pod \"d786e2d8-d256-456c-8885-56f246e71949\" (UID: \"d786e2d8-d256-456c-8885-56f246e71949\") "
Oct 30 12:53:25.715971 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:25.715940    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d786e2d8-d256-456c-8885-56f246e71949-maas-api-openshift-ca" pod "d786e2d8-d256-456c-8885-56f246e71949" (UID: "d786e2d8-d256-456c-8885-56f246e71949"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:25.717027 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:25.717002    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d786e2d8-d256-456c-8885-56f246e71949-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:25.718165 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:25.718133    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d786e2d8-d256-456c-8885-56f246e71949-maas-api-xks-ca" pod "d786e2d8-d256-456c-8885-56f246e71949" (UID: "d786e2d8-d256-456c-8885-56f246e71949"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:25.718387 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:25.718367    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d786e2d8-d256-456c-8885-56f246e71949-kube-api-access-8s4mq" pod "d786e2d8-d256-456c-8885-56f246e71949" (UID: "d786e2d8-d256-456c-8885-56f246e71949"). InnerVolumeSpecName "kube-api-access-8s4mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:25.818547 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:25.818519    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s4mq\" (UniqueName: \"kubernetes.io/projected/d786e2d8-d256-456c-8885-56f246e71949-kube-api-access-8s4mq\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:25.818547 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:25.818547    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d786e2d8-d256-456c-8885-56f246e71949-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:26.262147 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.262111    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"]
Oct 30 12:53:26.262786 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.262772    5837 cpu_manager.go:414] "containerMap: removing container" podUID="d786e2d8-d256-456c-8885-56f246e71949" containerName="revoke-keys"
Oct 30 12:53:26.262827 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.262787    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d786e2d8-d256-456c-8885-56f246e71949" containerName="revoke-keys"
Oct 30 12:53:26.262827 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.262801    5837 cpu_manager.go:414] "containerMap: removing container" podUID="c1755206-b5ef-4c8e-aeaa-d90b5317e36e" containerName="revoke-keys"
Oct 30 12:53:26.262827 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.262807    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1755206-b5ef-4c8e-aeaa-d90b5317e36e" containerName="revoke-keys"
Oct 30 12:53:26.262928 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.262902    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d786e2d8-d256-456c-8885-56f246e71949" containerName="revoke-keys"
Oct 30 12:53:26.262928 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.262913    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1755206-b5ef-4c8e-aeaa-d90b5317e36e" containerName="revoke-keys"
Oct 30 12:53:26.266838 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.266816    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"
Oct 30 12:53:26.286486 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.286460    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"]
Oct 30 12:53:26.325347 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.325319    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98bce3d3-8133-4ae8-a31a-e11d1a517495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47a349c3-jcb5c\" (UID: \"98bce3d3-8133-4ae8-a31a-e11d1a517495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"
Oct 30 12:53:26.325546 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.325530    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98bce3d3-8133-4ae8-a31a-e11d1a517495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47a349c3-jcb5c\" (UID: \"98bce3d3-8133-4ae8-a31a-e11d1a517495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"
Oct 30 12:53:26.325651 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.325636    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjd9d\" (UniqueName: \"kubernetes.io/projected/98bce3d3-8133-4ae8-a31a-e11d1a517495-kube-api-access-xjd9d\") pod \"maas-api-revoke-sub-47a349c3-jcb5c\" (UID: \"98bce3d3-8133-4ae8-a31a-e11d1a517495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"
Oct 30 12:53:26.426589 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.426559    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjd9d\" (UniqueName: \"kubernetes.io/projected/98bce3d3-8133-4ae8-a31a-e11d1a517495-kube-api-access-xjd9d\") pod \"maas-api-revoke-sub-47a349c3-jcb5c\" (UID: \"98bce3d3-8133-4ae8-a31a-e11d1a517495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"
Oct 30 12:53:26.426767 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.426652    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98bce3d3-8133-4ae8-a31a-e11d1a517495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47a349c3-jcb5c\" (UID: \"98bce3d3-8133-4ae8-a31a-e11d1a517495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"
Oct 30 12:53:26.426767 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.426733    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98bce3d3-8133-4ae8-a31a-e11d1a517495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47a349c3-jcb5c\" (UID: \"98bce3d3-8133-4ae8-a31a-e11d1a517495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"
Oct 30 12:53:26.427268 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.427249    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98bce3d3-8133-4ae8-a31a-e11d1a517495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47a349c3-jcb5c\" (UID: \"98bce3d3-8133-4ae8-a31a-e11d1a517495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"
Oct 30 12:53:26.429211 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.429190    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98bce3d3-8133-4ae8-a31a-e11d1a517495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47a349c3-jcb5c\" (UID: \"98bce3d3-8133-4ae8-a31a-e11d1a517495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"
Oct 30 12:53:26.444826 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.444799    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjd9d\" (UniqueName: \"kubernetes.io/projected/98bce3d3-8133-4ae8-a31a-e11d1a517495-kube-api-access-xjd9d\") pod \"maas-api-revoke-sub-47a349c3-jcb5c\" (UID: \"98bce3d3-8133-4ae8-a31a-e11d1a517495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"
Oct 30 12:53:26.460721 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.460702    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"
Oct 30 12:53:26.460826 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.460737    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps" event={"ID":"d786e2d8-d256-456c-8885-56f246e71949","Type":"ContainerDied","Data":"92501b6b9c51eac32393002ce28472853747cd4f99833a97df18134ab9181f7b"}
Oct 30 12:53:26.460826 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.460781    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92501b6b9c51eac32393002ce28472853747cd4f99833a97df18134ab9181f7b"
Oct 30 12:53:26.582984 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.582918    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"
Oct 30 12:53:26.880760 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:53:26.880730    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bce3d3_8133_4ae8_a31a_e11d1a517495.slice/crio-142f6e9703b9082893325f4ea21ad0fd3f9ae9e9e6318d4f68f8da9ac8ee7f90 WatchSource:0}: Error finding container 142f6e9703b9082893325f4ea21ad0fd3f9ae9e9e6318d4f68f8da9ac8ee7f90: Status 404 returned error can't find the container with id 142f6e9703b9082893325f4ea21ad0fd3f9ae9e9e6318d4f68f8da9ac8ee7f90
Oct 30 12:53:26.881598 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:26.881574    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"]
Oct 30 12:53:27.469773 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:27.469685    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c" event={"ID":"98bce3d3-8133-4ae8-a31a-e11d1a517495","Type":"ContainerStarted","Data":"f742bf746fec2f5d39068b088225e06c3f29d8331135f9b0c7306b656b8198b5"}
Oct 30 12:53:27.469937 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:27.469784    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c" event={"ID":"98bce3d3-8133-4ae8-a31a-e11d1a517495","Type":"ContainerStarted","Data":"142f6e9703b9082893325f4ea21ad0fd3f9ae9e9e6318d4f68f8da9ac8ee7f90"}
Oct 30 12:53:27.508695 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:27.508635    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c" podStartSLOduration=1.508619286 podStartE2EDuration="1.508619286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:27.5060452 +0000 UTC m=+996.030329419" watchObservedRunningTime="2026-10-30 12:53:27.508619286 +0000 UTC m=+996.032903507"
Oct 30 12:53:28.478116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:28.478082    5837 generic.go:356] "Generic (PLEG): container finished" podID="98bce3d3-8133-4ae8-a31a-e11d1a517495" containerID="f742bf746fec2f5d39068b088225e06c3f29d8331135f9b0c7306b656b8198b5" exitCode=0
Oct 30 12:53:28.478579 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:28.478176    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c" event={"ID":"98bce3d3-8133-4ae8-a31a-e11d1a517495","Type":"ContainerDied","Data":"f742bf746fec2f5d39068b088225e06c3f29d8331135f9b0c7306b656b8198b5"}
Oct 30 12:53:29.667926 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:29.667901    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"
Oct 30 12:53:29.774797 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:29.774714    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98bce3d3-8133-4ae8-a31a-e11d1a517495-kube-api-access-xjd9d\" (UniqueName: \"kubernetes.io/projected/98bce3d3-8133-4ae8-a31a-e11d1a517495-kube-api-access-xjd9d\") pod \"98bce3d3-8133-4ae8-a31a-e11d1a517495\" (UID: \"98bce3d3-8133-4ae8-a31a-e11d1a517495\") "
Oct 30 12:53:29.775007 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:29.774988    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98bce3d3-8133-4ae8-a31a-e11d1a517495-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98bce3d3-8133-4ae8-a31a-e11d1a517495-maas-api-xks-ca\") pod \"98bce3d3-8133-4ae8-a31a-e11d1a517495\" (UID: \"98bce3d3-8133-4ae8-a31a-e11d1a517495\") "
Oct 30 12:53:29.775091 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:29.775059    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98bce3d3-8133-4ae8-a31a-e11d1a517495-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98bce3d3-8133-4ae8-a31a-e11d1a517495-maas-api-openshift-ca\") pod \"98bce3d3-8133-4ae8-a31a-e11d1a517495\" (UID: \"98bce3d3-8133-4ae8-a31a-e11d1a517495\") "
Oct 30 12:53:29.775456 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:29.775433    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98bce3d3-8133-4ae8-a31a-e11d1a517495-maas-api-openshift-ca" pod "98bce3d3-8133-4ae8-a31a-e11d1a517495" (UID: "98bce3d3-8133-4ae8-a31a-e11d1a517495"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:29.775721 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:29.775685    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98bce3d3-8133-4ae8-a31a-e11d1a517495-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:29.777085 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:29.777051    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98bce3d3-8133-4ae8-a31a-e11d1a517495-kube-api-access-xjd9d" pod "98bce3d3-8133-4ae8-a31a-e11d1a517495" (UID: "98bce3d3-8133-4ae8-a31a-e11d1a517495"). InnerVolumeSpecName "kube-api-access-xjd9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:29.777497 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:29.777473    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98bce3d3-8133-4ae8-a31a-e11d1a517495-maas-api-xks-ca" pod "98bce3d3-8133-4ae8-a31a-e11d1a517495" (UID: "98bce3d3-8133-4ae8-a31a-e11d1a517495"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:29.876915 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:29.876884    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98bce3d3-8133-4ae8-a31a-e11d1a517495-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:29.876915 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:29.876910    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjd9d\" (UniqueName: \"kubernetes.io/projected/98bce3d3-8133-4ae8-a31a-e11d1a517495-kube-api-access-xjd9d\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:30.502379 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:30.502353    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"
Oct 30 12:53:30.502615 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:30.502382    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c" event={"ID":"98bce3d3-8133-4ae8-a31a-e11d1a517495","Type":"ContainerDied","Data":"142f6e9703b9082893325f4ea21ad0fd3f9ae9e9e6318d4f68f8da9ac8ee7f90"}
Oct 30 12:53:30.502615 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:30.502467    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="142f6e9703b9082893325f4ea21ad0fd3f9ae9e9e6318d4f68f8da9ac8ee7f90"
Oct 30 12:53:33.179741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.179657    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"]
Oct 30 12:53:33.180294 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.180281    5837 cpu_manager.go:414] "containerMap: removing container" podUID="98bce3d3-8133-4ae8-a31a-e11d1a517495" containerName="revoke-keys"
Oct 30 12:53:33.180342 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.180296    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98bce3d3-8133-4ae8-a31a-e11d1a517495" containerName="revoke-keys"
Oct 30 12:53:33.180436 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.180426    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="98bce3d3-8133-4ae8-a31a-e11d1a517495" containerName="revoke-keys"
Oct 30 12:53:33.184336 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.184318    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"
Oct 30 12:53:33.202017 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.201993    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"]
Oct 30 12:53:33.208880 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.208857    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:53:33.209011 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.208935    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:53:33.209011 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.208954    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:53:33.209747 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.209731    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:53:33.328038 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.328006    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49477b9-a4e4-4dad-9951-6153b32e1e3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7435d366-ghzhn\" (UID: \"c49477b9-a4e4-4dad-9951-6153b32e1e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"
Oct 30 12:53:33.328272 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.328252    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4nhn\" (UniqueName: \"kubernetes.io/projected/c49477b9-a4e4-4dad-9951-6153b32e1e3c-kube-api-access-h4nhn\") pod \"maas-api-revoke-sub-7435d366-ghzhn\" (UID: \"c49477b9-a4e4-4dad-9951-6153b32e1e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"
Oct 30 12:53:33.328377 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.328364    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49477b9-a4e4-4dad-9951-6153b32e1e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7435d366-ghzhn\" (UID: \"c49477b9-a4e4-4dad-9951-6153b32e1e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"
Oct 30 12:53:33.429724 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.429701    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4nhn\" (UniqueName: \"kubernetes.io/projected/c49477b9-a4e4-4dad-9951-6153b32e1e3c-kube-api-access-h4nhn\") pod \"maas-api-revoke-sub-7435d366-ghzhn\" (UID: \"c49477b9-a4e4-4dad-9951-6153b32e1e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"
Oct 30 12:53:33.429904 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.429789    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49477b9-a4e4-4dad-9951-6153b32e1e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7435d366-ghzhn\" (UID: \"c49477b9-a4e4-4dad-9951-6153b32e1e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"
Oct 30 12:53:33.429904 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.429835    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49477b9-a4e4-4dad-9951-6153b32e1e3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7435d366-ghzhn\" (UID: \"c49477b9-a4e4-4dad-9951-6153b32e1e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"
Oct 30 12:53:33.430630 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.430604    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49477b9-a4e4-4dad-9951-6153b32e1e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7435d366-ghzhn\" (UID: \"c49477b9-a4e4-4dad-9951-6153b32e1e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"
Oct 30 12:53:33.432301 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.432278    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49477b9-a4e4-4dad-9951-6153b32e1e3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7435d366-ghzhn\" (UID: \"c49477b9-a4e4-4dad-9951-6153b32e1e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"
Oct 30 12:53:33.463148 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.463125    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4nhn\" (UniqueName: \"kubernetes.io/projected/c49477b9-a4e4-4dad-9951-6153b32e1e3c-kube-api-access-h4nhn\") pod \"maas-api-revoke-sub-7435d366-ghzhn\" (UID: \"c49477b9-a4e4-4dad-9951-6153b32e1e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"
Oct 30 12:53:33.498318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.498289    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"
Oct 30 12:53:33.838905 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:33.838877    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"]
Oct 30 12:53:33.839036 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:53:33.838893    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49477b9_a4e4_4dad_9951_6153b32e1e3c.slice/crio-ca95aaed0378b9cf0ece252f981c28ebb97aebf406ea49c4d7b9c80178cc5b3f WatchSource:0}: Error finding container ca95aaed0378b9cf0ece252f981c28ebb97aebf406ea49c4d7b9c80178cc5b3f: Status 404 returned error can't find the container with id ca95aaed0378b9cf0ece252f981c28ebb97aebf406ea49c4d7b9c80178cc5b3f
Oct 30 12:53:34.533458 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:34.533419    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn" event={"ID":"c49477b9-a4e4-4dad-9951-6153b32e1e3c","Type":"ContainerStarted","Data":"af06b31008684dc78cb39f46122b9644f3da5b764b3d36502b4f712f0cf54a58"}
Oct 30 12:53:34.533852 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:34.533475    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn" event={"ID":"c49477b9-a4e4-4dad-9951-6153b32e1e3c","Type":"ContainerStarted","Data":"ca95aaed0378b9cf0ece252f981c28ebb97aebf406ea49c4d7b9c80178cc5b3f"}
Oct 30 12:53:34.574146 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:34.574089    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn" podStartSLOduration=1.5740747389999998 podStartE2EDuration="1.574074739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:34.571155522 +0000 UTC m=+1003.095439741" watchObservedRunningTime="2026-10-30 12:53:34.574074739 +0000 UTC m=+1003.098358958"
Oct 30 12:53:35.560781 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:35.560751    5837 generic.go:356] "Generic (PLEG): container finished" podID="c49477b9-a4e4-4dad-9951-6153b32e1e3c" containerID="af06b31008684dc78cb39f46122b9644f3da5b764b3d36502b4f712f0cf54a58" exitCode=0
Oct 30 12:53:35.561173 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:35.560828    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn" event={"ID":"c49477b9-a4e4-4dad-9951-6153b32e1e3c","Type":"ContainerDied","Data":"af06b31008684dc78cb39f46122b9644f3da5b764b3d36502b4f712f0cf54a58"}
Oct 30 12:53:36.768089 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.768043    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"]
Oct 30 12:53:36.774006 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.773976    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"
Oct 30 12:53:36.792951 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.792923    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"]
Oct 30 12:53:36.824761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.824736    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"
Oct 30 12:53:36.889075 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.888995    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-750ae661-p9zwl\" (UID: \"6b6c89d9-9c3c-4f41-9345-104dc6adbd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"
Oct 30 12:53:36.889075 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.889051    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-750ae661-p9zwl\" (UID: \"6b6c89d9-9c3c-4f41-9345-104dc6adbd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"
Oct 30 12:53:36.889305 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.889270    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9mg8\" (UniqueName: \"kubernetes.io/projected/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-kube-api-access-d9mg8\") pod \"maas-api-revoke-sub-750ae661-p9zwl\" (UID: \"6b6c89d9-9c3c-4f41-9345-104dc6adbd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"
Oct 30 12:53:36.990453 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.990424    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c49477b9-a4e4-4dad-9951-6153b32e1e3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49477b9-a4e4-4dad-9951-6153b32e1e3c-maas-api-xks-ca\") pod \"c49477b9-a4e4-4dad-9951-6153b32e1e3c\" (UID: \"c49477b9-a4e4-4dad-9951-6153b32e1e3c\") "
Oct 30 12:53:36.990614 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.990496    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c49477b9-a4e4-4dad-9951-6153b32e1e3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49477b9-a4e4-4dad-9951-6153b32e1e3c-maas-api-openshift-ca\") pod \"c49477b9-a4e4-4dad-9951-6153b32e1e3c\" (UID: \"c49477b9-a4e4-4dad-9951-6153b32e1e3c\") "
Oct 30 12:53:36.990681 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.990668    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49477b9-a4e4-4dad-9951-6153b32e1e3c-kube-api-access-h4nhn\" (UniqueName: \"kubernetes.io/projected/c49477b9-a4e4-4dad-9951-6153b32e1e3c-kube-api-access-h4nhn\") pod \"c49477b9-a4e4-4dad-9951-6153b32e1e3c\" (UID: \"c49477b9-a4e4-4dad-9951-6153b32e1e3c\") "
Oct 30 12:53:36.990862 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.990833    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49477b9-a4e4-4dad-9951-6153b32e1e3c-maas-api-openshift-ca" pod "c49477b9-a4e4-4dad-9951-6153b32e1e3c" (UID: "c49477b9-a4e4-4dad-9951-6153b32e1e3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:36.991064 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.991041    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9mg8\" (UniqueName: \"kubernetes.io/projected/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-kube-api-access-d9mg8\") pod \"maas-api-revoke-sub-750ae661-p9zwl\" (UID: \"6b6c89d9-9c3c-4f41-9345-104dc6adbd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"
Oct 30 12:53:36.991165 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.991131    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-750ae661-p9zwl\" (UID: \"6b6c89d9-9c3c-4f41-9345-104dc6adbd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"
Oct 30 12:53:36.991349 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.991331    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-750ae661-p9zwl\" (UID: \"6b6c89d9-9c3c-4f41-9345-104dc6adbd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"
Oct 30 12:53:36.991596 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.991577    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49477b9-a4e4-4dad-9951-6153b32e1e3c-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:36.992148 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.992124    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-750ae661-p9zwl\" (UID: \"6b6c89d9-9c3c-4f41-9345-104dc6adbd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"
Oct 30 12:53:36.993011 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.992989    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c49477b9-a4e4-4dad-9951-6153b32e1e3c-maas-api-xks-ca" pod "c49477b9-a4e4-4dad-9951-6153b32e1e3c" (UID: "c49477b9-a4e4-4dad-9951-6153b32e1e3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:36.993011 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.992988    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49477b9-a4e4-4dad-9951-6153b32e1e3c-kube-api-access-h4nhn" pod "c49477b9-a4e4-4dad-9951-6153b32e1e3c" (UID: "c49477b9-a4e4-4dad-9951-6153b32e1e3c"). InnerVolumeSpecName "kube-api-access-h4nhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:36.993564 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:36.993548    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-750ae661-p9zwl\" (UID: \"6b6c89d9-9c3c-4f41-9345-104dc6adbd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"
Oct 30 12:53:37.009987 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:37.009957    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9mg8\" (UniqueName: \"kubernetes.io/projected/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-kube-api-access-d9mg8\") pod \"maas-api-revoke-sub-750ae661-p9zwl\" (UID: \"6b6c89d9-9c3c-4f41-9345-104dc6adbd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"
Oct 30 12:53:37.090754 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:37.090725    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"
Oct 30 12:53:37.092497 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:37.092474    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4nhn\" (UniqueName: \"kubernetes.io/projected/c49477b9-a4e4-4dad-9951-6153b32e1e3c-kube-api-access-h4nhn\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:37.092497 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:37.092499    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49477b9-a4e4-4dad-9951-6153b32e1e3c-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:37.408810 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:37.408780    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"]
Oct 30 12:53:37.409362 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:53:37.409336    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6c89d9_9c3c_4f41_9345_104dc6adbd67.slice/crio-ab5ce9c17c02aa7c2df6dedb9b6e7420a8b81947682b98b00ede2c043161df64 WatchSource:0}: Error finding container ab5ce9c17c02aa7c2df6dedb9b6e7420a8b81947682b98b00ede2c043161df64: Status 404 returned error can't find the container with id ab5ce9c17c02aa7c2df6dedb9b6e7420a8b81947682b98b00ede2c043161df64
Oct 30 12:53:37.657931 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:37.657850    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"
Oct 30 12:53:37.657931 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:37.657887    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn" event={"ID":"c49477b9-a4e4-4dad-9951-6153b32e1e3c","Type":"ContainerDied","Data":"ca95aaed0378b9cf0ece252f981c28ebb97aebf406ea49c4d7b9c80178cc5b3f"}
Oct 30 12:53:37.658131 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:37.657942    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca95aaed0378b9cf0ece252f981c28ebb97aebf406ea49c4d7b9c80178cc5b3f"
Oct 30 12:53:37.662986 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:37.662960    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl" event={"ID":"6b6c89d9-9c3c-4f41-9345-104dc6adbd67","Type":"ContainerStarted","Data":"ab5ce9c17c02aa7c2df6dedb9b6e7420a8b81947682b98b00ede2c043161df64"}
Oct 30 12:53:38.673216 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:38.673184    5837 generic.go:356] "Generic (PLEG): container finished" podID="6b6c89d9-9c3c-4f41-9345-104dc6adbd67" containerID="f3e75fcb18b0116e5ad2cc8d8622314b02c61a450075da6cf4889e02883be67e" exitCode=0
Oct 30 12:53:38.673667 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:38.673276    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl" event={"ID":"6b6c89d9-9c3c-4f41-9345-104dc6adbd67","Type":"ContainerDied","Data":"f3e75fcb18b0116e5ad2cc8d8622314b02c61a450075da6cf4889e02883be67e"}
Oct 30 12:53:39.849322 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:39.849294    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"
Oct 30 12:53:39.933011 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:39.932942    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-maas-api-xks-ca\") pod \"6b6c89d9-9c3c-4f41-9345-104dc6adbd67\" (UID: \"6b6c89d9-9c3c-4f41-9345-104dc6adbd67\") "
Oct 30 12:53:39.933167 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:39.933154    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-kube-api-access-d9mg8\" (UniqueName: \"kubernetes.io/projected/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-kube-api-access-d9mg8\") pod \"6b6c89d9-9c3c-4f41-9345-104dc6adbd67\" (UID: \"6b6c89d9-9c3c-4f41-9345-104dc6adbd67\") "
Oct 30 12:53:39.933213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:39.933191    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-maas-api-openshift-ca\") pod \"6b6c89d9-9c3c-4f41-9345-104dc6adbd67\" (UID: \"6b6c89d9-9c3c-4f41-9345-104dc6adbd67\") "
Oct 30 12:53:39.933594 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:39.933567    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-maas-api-openshift-ca" pod "6b6c89d9-9c3c-4f41-9345-104dc6adbd67" (UID: "6b6c89d9-9c3c-4f41-9345-104dc6adbd67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:39.935165 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:39.935142    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-maas-api-xks-ca" pod "6b6c89d9-9c3c-4f41-9345-104dc6adbd67" (UID: "6b6c89d9-9c3c-4f41-9345-104dc6adbd67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:39.935240 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:39.935221    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-kube-api-access-d9mg8" pod "6b6c89d9-9c3c-4f41-9345-104dc6adbd67" (UID: "6b6c89d9-9c3c-4f41-9345-104dc6adbd67"). InnerVolumeSpecName "kube-api-access-d9mg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:40.033200 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:40.033171    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"]
Oct 30 12:53:40.034585 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:40.034568    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9mg8\" (UniqueName: \"kubernetes.io/projected/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-kube-api-access-d9mg8\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:40.034585 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:40.034585    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:40.034715 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:40.034595    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6c89d9-9c3c-4f41-9345-104dc6adbd67-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:40.036940 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:40.036917    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e861da0-4gkmk"]
Oct 30 12:53:40.689820 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:40.689791    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"
Oct 30 12:53:40.689994 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:40.689822    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl" event={"ID":"6b6c89d9-9c3c-4f41-9345-104dc6adbd67","Type":"ContainerDied","Data":"ab5ce9c17c02aa7c2df6dedb9b6e7420a8b81947682b98b00ede2c043161df64"}
Oct 30 12:53:40.689994 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:40.689872    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab5ce9c17c02aa7c2df6dedb9b6e7420a8b81947682b98b00ede2c043161df64"
Oct 30 12:53:41.606116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:41.606082    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99188a77-6f30-45a7-b018-04f6288901e1" path="/var/lib/kubelet/pods/99188a77-6f30-45a7-b018-04f6288901e1/volumes"
Oct 30 12:53:45.094650 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:53:45.094618    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:53:45.963756 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:45.963715    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"]
Oct 30 12:53:45.965002 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:45.964972    5837 cpu_manager.go:414] "containerMap: removing container" podUID="6b6c89d9-9c3c-4f41-9345-104dc6adbd67" containerName="revoke-keys"
Oct 30 12:53:45.965002 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:45.965000    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6c89d9-9c3c-4f41-9345-104dc6adbd67" containerName="revoke-keys"
Oct 30 12:53:45.965206 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:45.965081    5837 cpu_manager.go:414] "containerMap: removing container" podUID="c49477b9-a4e4-4dad-9951-6153b32e1e3c" containerName="revoke-keys"
Oct 30 12:53:45.965206 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:45.965090    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49477b9-a4e4-4dad-9951-6153b32e1e3c" containerName="revoke-keys"
Oct 30 12:53:45.965313 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:45.965255    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49477b9-a4e4-4dad-9951-6153b32e1e3c" containerName="revoke-keys"
Oct 30 12:53:45.965313 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:45.965274    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6c89d9-9c3c-4f41-9345-104dc6adbd67" containerName="revoke-keys"
Oct 30 12:53:45.970263 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:45.970238    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:45.993618 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:45.993586    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"]
Oct 30 12:53:45.998670 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:45.998635    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7a04e002-openshift-default-dockercfg-vxgwf\""
Oct 30 12:53:46.045700 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.045658    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"]
Oct 30 12:53:46.051533 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.051496    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2f7789-trb8v"]
Oct 30 12:53:46.124538 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.124499    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-credential-socket\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.124909 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.124565    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-podinfo\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.124909 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.124690    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-workload-socket\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.124909 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.124720    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-token\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.124909 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.124886    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-workload-certs\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.125037 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.124923    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istiod-ca-cert\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.125037 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.124941    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z4hq\" (UniqueName: \"kubernetes.io/projected/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-kube-api-access-4z4hq\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.125037 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.125020    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-data\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.125129 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.125046    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-envoy\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.230586 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.230541    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-envoy\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.231368 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.231341    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-credential-socket\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.232029 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.232011    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-podinfo\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.232203 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.232183    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-workload-socket\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.232753 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.232734    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-token\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.233124 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.233106    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-workload-certs\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.233259 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.233243    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istiod-ca-cert\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.233355 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.233343    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z4hq\" (UniqueName: \"kubernetes.io/projected/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-kube-api-access-4z4hq\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.233528 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.233511    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-data\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.233998 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.233979    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-data\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.234128 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.234098    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-workload-socket\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.234193 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.231934    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-credential-socket\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.234509 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.234492    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-workload-certs\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.234924 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.234865    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istiod-ca-cert\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.243459 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.242119    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-podinfo\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.247751 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.247699    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-envoy\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.261247 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.261182    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-token\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.264282 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.264256    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z4hq\" (UniqueName: \"kubernetes.io/projected/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-kube-api-access-4z4hq\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-92fw5\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.292142 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.290428    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:46.640646 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:53:46.640614    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a3fc92_01d1_45c0_ab72_c1eb52ed7113.slice/crio-1079b3d6ea5d2b3e475e2b75dea7b9b723ab917dc7ea8f9bad341ac240b48390 WatchSource:0}: Error finding container 1079b3d6ea5d2b3e475e2b75dea7b9b723ab917dc7ea8f9bad341ac240b48390: Status 404 returned error can't find the container with id 1079b3d6ea5d2b3e475e2b75dea7b9b723ab917dc7ea8f9bad341ac240b48390
Oct 30 12:53:46.640906 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.640876    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"]
Oct 30 12:53:46.643307 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.643275    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:46.643385 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.643337    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:46.643385 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.643366    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:46.747266 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.747226    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5" event={"ID":"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113","Type":"ContainerStarted","Data":"b92a811e43f026fa57e988fd33f822abda88ba9b84353a6cc22cdbf3738933f9"}
Oct 30 12:53:46.747484 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.747293    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5" event={"ID":"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113","Type":"ContainerStarted","Data":"1079b3d6ea5d2b3e475e2b75dea7b9b723ab917dc7ea8f9bad341ac240b48390"}
Oct 30 12:53:46.796214 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:46.796161    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5" podStartSLOduration=1.79614741 podStartE2EDuration="1.79614741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:46.795837412 +0000 UTC m=+1015.320121648" watchObservedRunningTime="2026-10-30 12:53:46.79614741 +0000 UTC m=+1015.320431628"
Oct 30 12:53:47.290951 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:47.290907    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:47.296698 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:47.296674    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:47.627541 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:47.627427    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="974423f8-8df0-4991-bde0-d09656d03d89" path="/var/lib/kubelet/pods/974423f8-8df0-4991-bde0-d09656d03d89/volumes"
Oct 30 12:53:47.755340 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:47.755315    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:47.756370 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:47.756350    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:53:55.179063 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.179026    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"]
Oct 30 12:53:55.184993 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.184973    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"
Oct 30 12:53:55.203322 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.203291    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"]
Oct 30 12:53:55.210132 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.210104    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:53:55.210132 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.210102    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:53:55.210300 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.210155    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:53:55.210300 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.210293    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:53:55.273992 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.273964    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvffx\" (UniqueName: \"kubernetes.io/projected/c78a3d85-7386-47c5-9eb1-c7faea052546-kube-api-access-cvffx\") pod \"maas-api-revoke-sub-32501fc6-sls9l\" (UID: \"c78a3d85-7386-47c5-9eb1-c7faea052546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"
Oct 30 12:53:55.274138 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.274002    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78a3d85-7386-47c5-9eb1-c7faea052546-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32501fc6-sls9l\" (UID: \"c78a3d85-7386-47c5-9eb1-c7faea052546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"
Oct 30 12:53:55.274138 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.274043    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78a3d85-7386-47c5-9eb1-c7faea052546-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32501fc6-sls9l\" (UID: \"c78a3d85-7386-47c5-9eb1-c7faea052546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"
Oct 30 12:53:55.375589 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.375550    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvffx\" (UniqueName: \"kubernetes.io/projected/c78a3d85-7386-47c5-9eb1-c7faea052546-kube-api-access-cvffx\") pod \"maas-api-revoke-sub-32501fc6-sls9l\" (UID: \"c78a3d85-7386-47c5-9eb1-c7faea052546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"
Oct 30 12:53:55.375749 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.375609    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78a3d85-7386-47c5-9eb1-c7faea052546-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32501fc6-sls9l\" (UID: \"c78a3d85-7386-47c5-9eb1-c7faea052546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"
Oct 30 12:53:55.375749 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.375666    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78a3d85-7386-47c5-9eb1-c7faea052546-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32501fc6-sls9l\" (UID: \"c78a3d85-7386-47c5-9eb1-c7faea052546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"
Oct 30 12:53:55.376282 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.376258    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78a3d85-7386-47c5-9eb1-c7faea052546-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32501fc6-sls9l\" (UID: \"c78a3d85-7386-47c5-9eb1-c7faea052546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"
Oct 30 12:53:55.378149 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.378129    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78a3d85-7386-47c5-9eb1-c7faea052546-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32501fc6-sls9l\" (UID: \"c78a3d85-7386-47c5-9eb1-c7faea052546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"
Oct 30 12:53:55.394381 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.394351    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvffx\" (UniqueName: \"kubernetes.io/projected/c78a3d85-7386-47c5-9eb1-c7faea052546-kube-api-access-cvffx\") pod \"maas-api-revoke-sub-32501fc6-sls9l\" (UID: \"c78a3d85-7386-47c5-9eb1-c7faea052546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"
Oct 30 12:53:55.498891 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.498868    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"
Oct 30 12:53:55.795300 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.795276    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"]
Oct 30 12:53:55.795886 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:53:55.795862    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78a3d85_7386_47c5_9eb1_c7faea052546.slice/crio-95058fd5dadaf9ff38b0581ccb80359e196267b2a08b89d9e0f9cbfa42cd02f8 WatchSource:0}: Error finding container 95058fd5dadaf9ff38b0581ccb80359e196267b2a08b89d9e0f9cbfa42cd02f8: Status 404 returned error can't find the container with id 95058fd5dadaf9ff38b0581ccb80359e196267b2a08b89d9e0f9cbfa42cd02f8
Oct 30 12:53:55.850836 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:55.850811    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l" event={"ID":"c78a3d85-7386-47c5-9eb1-c7faea052546","Type":"ContainerStarted","Data":"95058fd5dadaf9ff38b0581ccb80359e196267b2a08b89d9e0f9cbfa42cd02f8"}
Oct 30 12:53:56.861553 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:56.861522    5837 generic.go:356] "Generic (PLEG): container finished" podID="c78a3d85-7386-47c5-9eb1-c7faea052546" containerID="768613622ff22d520b5f7d268babd930840d2541020e674d253530cff8c7fb6c" exitCode=0
Oct 30 12:53:56.861913 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:56.861600    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l" event={"ID":"c78a3d85-7386-47c5-9eb1-c7faea052546","Type":"ContainerDied","Data":"768613622ff22d520b5f7d268babd930840d2541020e674d253530cff8c7fb6c"}
Oct 30 12:53:57.928769 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:57.928726    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"]
Oct 30 12:53:57.935424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:57.934098    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:57.960470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:57.960422    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"]
Oct 30 12:53:57.966451 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:57.966426    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0a531c-openshift-default-dockercfg-qscl4\""
Oct 30 12:53:58.137654 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.137616    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-workload-socket\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.137904 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.137888    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-workload-certs\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.138086 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.138071    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-credential-socket\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.138361 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.138344    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-istio-envoy\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.138515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.138501    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484f0dd9-59d8-4690-a051-044c1fed4945-istiod-ca-cert\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.138645 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.138632    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484f0dd9-59d8-4690-a051-044c1fed4945-istio-token\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.138752 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.138738    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzgsm\" (UniqueName: \"kubernetes.io/projected/484f0dd9-59d8-4690-a051-044c1fed4945-kube-api-access-vzgsm\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.138867 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.138856    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-istio-data\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.139004 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.138991    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484f0dd9-59d8-4690-a051-044c1fed4945-istio-podinfo\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.241887 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.241846    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-istio-envoy\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.242123 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.242105    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484f0dd9-59d8-4690-a051-044c1fed4945-istiod-ca-cert\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.242274 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.242261    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484f0dd9-59d8-4690-a051-044c1fed4945-istio-token\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.242416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.242383    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzgsm\" (UniqueName: \"kubernetes.io/projected/484f0dd9-59d8-4690-a051-044c1fed4945-kube-api-access-vzgsm\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.242560 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.242545    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-istio-data\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.242728 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.242700    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484f0dd9-59d8-4690-a051-044c1fed4945-istio-podinfo\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.242985 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.242970    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-workload-socket\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.243103 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.243091    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-workload-certs\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.243275 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.243262    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-credential-socket\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.244362 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.244337    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-istio-data\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.245776 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.245759    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-credential-socket\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.246780 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.246762    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484f0dd9-59d8-4690-a051-044c1fed4945-istiod-ca-cert\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.247922 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.247550    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-workload-certs\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.248610 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.248586    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-workload-socket\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.272416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.268485    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484f0dd9-59d8-4690-a051-044c1fed4945-istio-token\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.272704 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.272651    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzgsm\" (UniqueName: \"kubernetes.io/projected/484f0dd9-59d8-4690-a051-044c1fed4945-kube-api-access-vzgsm\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.273094 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.273069    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484f0dd9-59d8-4690-a051-044c1fed4945-istio-podinfo\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.273278 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.273259    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-istio-envoy\") pod \"e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") " pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.278733 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.278528    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:53:58.567467 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.564604    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"
Oct 30 12:53:58.592446 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.589929    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c78a3d85-7386-47c5-9eb1-c7faea052546-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78a3d85-7386-47c5-9eb1-c7faea052546-maas-api-openshift-ca\") pod \"c78a3d85-7386-47c5-9eb1-c7faea052546\" (UID: \"c78a3d85-7386-47c5-9eb1-c7faea052546\") "
Oct 30 12:53:58.592446 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.590125    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c78a3d85-7386-47c5-9eb1-c7faea052546-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78a3d85-7386-47c5-9eb1-c7faea052546-maas-api-xks-ca\") pod \"c78a3d85-7386-47c5-9eb1-c7faea052546\" (UID: \"c78a3d85-7386-47c5-9eb1-c7faea052546\") "
Oct 30 12:53:58.592446 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.590264    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c78a3d85-7386-47c5-9eb1-c7faea052546-kube-api-access-cvffx\" (UniqueName: \"kubernetes.io/projected/c78a3d85-7386-47c5-9eb1-c7faea052546-kube-api-access-cvffx\") pod \"c78a3d85-7386-47c5-9eb1-c7faea052546\" (UID: \"c78a3d85-7386-47c5-9eb1-c7faea052546\") "
Oct 30 12:53:58.599679 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.599640    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78a3d85-7386-47c5-9eb1-c7faea052546-maas-api-openshift-ca" pod "c78a3d85-7386-47c5-9eb1-c7faea052546" (UID: "c78a3d85-7386-47c5-9eb1-c7faea052546"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:58.606419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.603830    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c78a3d85-7386-47c5-9eb1-c7faea052546-maas-api-xks-ca" pod "c78a3d85-7386-47c5-9eb1-c7faea052546" (UID: "c78a3d85-7386-47c5-9eb1-c7faea052546"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:58.606761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.606730    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78a3d85-7386-47c5-9eb1-c7faea052546-kube-api-access-cvffx" pod "c78a3d85-7386-47c5-9eb1-c7faea052546" (UID: "c78a3d85-7386-47c5-9eb1-c7faea052546"). InnerVolumeSpecName "kube-api-access-cvffx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:58.700509 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.700481    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78a3d85-7386-47c5-9eb1-c7faea052546-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:58.700681 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.700665    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78a3d85-7386-47c5-9eb1-c7faea052546-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:58.700779 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.700766    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvffx\" (UniqueName: \"kubernetes.io/projected/c78a3d85-7386-47c5-9eb1-c7faea052546-kube-api-access-cvffx\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:53:58.949751 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.949647    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l" event={"ID":"c78a3d85-7386-47c5-9eb1-c7faea052546","Type":"ContainerDied","Data":"95058fd5dadaf9ff38b0581ccb80359e196267b2a08b89d9e0f9cbfa42cd02f8"}
Oct 30 12:53:58.950231 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.949753    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95058fd5dadaf9ff38b0581ccb80359e196267b2a08b89d9e0f9cbfa42cd02f8"
Oct 30 12:53:58.950231 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.949828    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"
Oct 30 12:53:58.982274 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.982231    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:58.982578 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.982552    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:58.983246 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:58.983223    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:59.003730 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:53:59.003705    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"]
Oct 30 12:54:00.006143 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:00.006100    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" event={"ID":"484f0dd9-59d8-4690-a051-044c1fed4945","Type":"ContainerStarted","Data":"85593eac598b59dc3265222df1e10753f93785187b65e244934970880951cd99"}
Oct 30 12:54:00.006881 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:00.006839    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" event={"ID":"484f0dd9-59d8-4690-a051-044c1fed4945","Type":"ContainerStarted","Data":"032ce4df6a308efca01feb283be3780abb6babdd4c09768d42930e5b0fa733c0"}
Oct 30 12:54:00.107423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:00.103444    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" podStartSLOduration=3.103420202 podStartE2EDuration="3.103420202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:00.092336984 +0000 UTC m=+1028.616621204" watchObservedRunningTime="2026-10-30 12:54:00.103420202 +0000 UTC m=+1028.627704422"
Oct 30 12:54:00.282636 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:00.282547    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:54:00.282948 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:00.282924    5837 patch_prober.go:29] interesting pod/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 12:54:00.283129 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:00.283099    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 12:54:00.377582 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:00.377536    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:00.472622 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:00.472564    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:01.288717 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:01.288527    5837 patch_prober.go:29] interesting pod/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 12:54:01.288717 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:01.288593    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 12:54:02.037541 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:02.037489    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:02.407479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:02.403498    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:54:02.661503 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:02.661384    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:54:02.725432 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:02.701711    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:54:03.153423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:03.086106    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:03.153423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:03.108797    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:54:03.426533 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:03.425256    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:03.430955 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:03.430924    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:54:03.723538 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:03.718253    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:03.988535 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:03.988498    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:54:03.996510 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:03.996485    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:54:03.996876 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:03.996857    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:54:04.089258 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:04.089215    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:04.430511 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:04.427577    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:04.430511 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:04.428513    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:04.790725 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:04.790683    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:54:04.969553 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:04.969513    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:54:04.969859 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:04.969824    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" podUID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:05.110587 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:05.110486    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:54:05.436526 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:05.435524    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:54:05.800419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:05.799477    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:05.943034 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:05.943001    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:54:06.442237 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:06.442191    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:06.812470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:06.810942    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:06.944698 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:06.944605    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:07.449644 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:07.449599    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:54:07.593667 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:07.593637    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:54:07.661416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:07.658855    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:54:11.028410 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:11.028348    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"]
Oct 30 12:54:11.039998 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:11.039966    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"]
Oct 30 12:54:11.043779 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:11.043759    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590a9ce4-pwh6z"]
Oct 30 12:54:11.047002 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:11.046979    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dfac4a7-2rkb6"]
Oct 30 12:54:11.606207 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:11.606172    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f61b3a-ae4c-4506-b9ff-d95bce390791" path="/var/lib/kubelet/pods/34f61b3a-ae4c-4506-b9ff-d95bce390791/volumes"
Oct 30 12:54:11.606699 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:11.606685    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71746b8c-2788-4d44-bbce-10fea0de7d57" path="/var/lib/kubelet/pods/71746b8c-2788-4d44-bbce-10fea0de7d57/volumes"
Oct 30 12:54:13.617329 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.617297    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"]
Oct 30 12:54:13.617964 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.617950    5837 cpu_manager.go:414] "containerMap: removing container" podUID="c78a3d85-7386-47c5-9eb1-c7faea052546" containerName="revoke-keys"
Oct 30 12:54:13.618009 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.617965    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c78a3d85-7386-47c5-9eb1-c7faea052546" containerName="revoke-keys"
Oct 30 12:54:13.618104 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.618094    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="c78a3d85-7386-47c5-9eb1-c7faea052546" containerName="revoke-keys"
Oct 30 12:54:13.622890 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.622873    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"
Oct 30 12:54:13.636844 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.636813    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:54:13.638839 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.638817    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"]
Oct 30 12:54:13.646202 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.646147    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:54:13.646360 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.646333    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:54:13.648583 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.648561    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:54:13.694939 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.694910    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13ea519-9c63-4fb4-abc1-98f4eca8b218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ee7e86-zf5vh\" (UID: \"c13ea519-9c63-4fb4-abc1-98f4eca8b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"
Oct 30 12:54:13.695056 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.694978    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13ea519-9c63-4fb4-abc1-98f4eca8b218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ee7e86-zf5vh\" (UID: \"c13ea519-9c63-4fb4-abc1-98f4eca8b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"
Oct 30 12:54:13.695199 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.695182    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n54gf\" (UniqueName: \"kubernetes.io/projected/c13ea519-9c63-4fb4-abc1-98f4eca8b218-kube-api-access-n54gf\") pod \"maas-api-revoke-sub-c1ee7e86-zf5vh\" (UID: \"c13ea519-9c63-4fb4-abc1-98f4eca8b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"
Oct 30 12:54:13.796174 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.796143    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13ea519-9c63-4fb4-abc1-98f4eca8b218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ee7e86-zf5vh\" (UID: \"c13ea519-9c63-4fb4-abc1-98f4eca8b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"
Oct 30 12:54:13.796356 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.796212    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13ea519-9c63-4fb4-abc1-98f4eca8b218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ee7e86-zf5vh\" (UID: \"c13ea519-9c63-4fb4-abc1-98f4eca8b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"
Oct 30 12:54:13.796356 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.796279    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54gf\" (UniqueName: \"kubernetes.io/projected/c13ea519-9c63-4fb4-abc1-98f4eca8b218-kube-api-access-n54gf\") pod \"maas-api-revoke-sub-c1ee7e86-zf5vh\" (UID: \"c13ea519-9c63-4fb4-abc1-98f4eca8b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"
Oct 30 12:54:13.796967 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.796942    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13ea519-9c63-4fb4-abc1-98f4eca8b218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ee7e86-zf5vh\" (UID: \"c13ea519-9c63-4fb4-abc1-98f4eca8b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"
Oct 30 12:54:13.799048 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.799029    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13ea519-9c63-4fb4-abc1-98f4eca8b218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ee7e86-zf5vh\" (UID: \"c13ea519-9c63-4fb4-abc1-98f4eca8b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"
Oct 30 12:54:13.815468 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.815446    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n54gf\" (UniqueName: \"kubernetes.io/projected/c13ea519-9c63-4fb4-abc1-98f4eca8b218-kube-api-access-n54gf\") pod \"maas-api-revoke-sub-c1ee7e86-zf5vh\" (UID: \"c13ea519-9c63-4fb4-abc1-98f4eca8b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"
Oct 30 12:54:13.937052 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:13.936967    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"
Oct 30 12:54:14.238077 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:14.238049    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"]
Oct 30 12:54:14.238196 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:54:14.238089    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13ea519_9c63_4fb4_abc1_98f4eca8b218.slice/crio-6bf5c65974df730df5897608f366c7f3b0b97f0399496714468d02edb337281b WatchSource:0}: Error finding container 6bf5c65974df730df5897608f366c7f3b0b97f0399496714468d02edb337281b: Status 404 returned error can't find the container with id 6bf5c65974df730df5897608f366c7f3b0b97f0399496714468d02edb337281b
Oct 30 12:54:15.031752 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:15.031706    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh" event={"ID":"c13ea519-9c63-4fb4-abc1-98f4eca8b218","Type":"ContainerStarted","Data":"657f28bd59b471d3c024f4e30d6d96afb870f5d6296ca998c1c2fea388d94890"}
Oct 30 12:54:15.032133 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:15.031816    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh" event={"ID":"c13ea519-9c63-4fb4-abc1-98f4eca8b218","Type":"ContainerStarted","Data":"6bf5c65974df730df5897608f366c7f3b0b97f0399496714468d02edb337281b"}
Oct 30 12:54:15.036288 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:15.036260    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"]
Oct 30 12:54:15.047263 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:15.047238    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c2ec46-fdxbn"]
Oct 30 12:54:15.072923 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:15.072879    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh" podStartSLOduration=2.072867296 podStartE2EDuration="2.072867296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:15.06913601 +0000 UTC m=+1043.593420230" watchObservedRunningTime="2026-10-30 12:54:15.072867296 +0000 UTC m=+1043.597151515"
Oct 30 12:54:15.096117 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:54:15.096084    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:54:15.605722 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:15.605690    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d644d9-84a3-4f85-96f2-417a7e7ae36c" path="/var/lib/kubelet/pods/93d644d9-84a3-4f85-96f2-417a7e7ae36c/volumes"
Oct 30 12:54:16.044138 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:16.044103    5837 generic.go:356] "Generic (PLEG): container finished" podID="c13ea519-9c63-4fb4-abc1-98f4eca8b218" containerID="657f28bd59b471d3c024f4e30d6d96afb870f5d6296ca998c1c2fea388d94890" exitCode=0
Oct 30 12:54:16.044138 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:16.044152    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh" event={"ID":"c13ea519-9c63-4fb4-abc1-98f4eca8b218","Type":"ContainerDied","Data":"657f28bd59b471d3c024f4e30d6d96afb870f5d6296ca998c1c2fea388d94890"}
Oct 30 12:54:17.210687 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:17.210662    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"
Oct 30 12:54:17.242677 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:17.242653    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13ea519-9c63-4fb4-abc1-98f4eca8b218-kube-api-access-n54gf\" (UniqueName: \"kubernetes.io/projected/c13ea519-9c63-4fb4-abc1-98f4eca8b218-kube-api-access-n54gf\") pod \"c13ea519-9c63-4fb4-abc1-98f4eca8b218\" (UID: \"c13ea519-9c63-4fb4-abc1-98f4eca8b218\") "
Oct 30 12:54:17.242792 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:17.242721    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13ea519-9c63-4fb4-abc1-98f4eca8b218-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13ea519-9c63-4fb4-abc1-98f4eca8b218-maas-api-openshift-ca\") pod \"c13ea519-9c63-4fb4-abc1-98f4eca8b218\" (UID: \"c13ea519-9c63-4fb4-abc1-98f4eca8b218\") "
Oct 30 12:54:17.242949 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:17.242936    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c13ea519-9c63-4fb4-abc1-98f4eca8b218-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13ea519-9c63-4fb4-abc1-98f4eca8b218-maas-api-xks-ca\") pod \"c13ea519-9c63-4fb4-abc1-98f4eca8b218\" (UID: \"c13ea519-9c63-4fb4-abc1-98f4eca8b218\") "
Oct 30 12:54:17.243107 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:17.243070    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13ea519-9c63-4fb4-abc1-98f4eca8b218-maas-api-openshift-ca" pod "c13ea519-9c63-4fb4-abc1-98f4eca8b218" (UID: "c13ea519-9c63-4fb4-abc1-98f4eca8b218"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:54:17.243573 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:17.243476    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13ea519-9c63-4fb4-abc1-98f4eca8b218-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:54:17.245222 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:17.245194    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13ea519-9c63-4fb4-abc1-98f4eca8b218-maas-api-xks-ca" pod "c13ea519-9c63-4fb4-abc1-98f4eca8b218" (UID: "c13ea519-9c63-4fb4-abc1-98f4eca8b218"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:54:17.245530 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:17.245507    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13ea519-9c63-4fb4-abc1-98f4eca8b218-kube-api-access-n54gf" pod "c13ea519-9c63-4fb4-abc1-98f4eca8b218" (UID: "c13ea519-9c63-4fb4-abc1-98f4eca8b218"). InnerVolumeSpecName "kube-api-access-n54gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:54:17.344634 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:17.344560    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13ea519-9c63-4fb4-abc1-98f4eca8b218-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:54:17.344634 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:17.344589    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n54gf\" (UniqueName: \"kubernetes.io/projected/c13ea519-9c63-4fb4-abc1-98f4eca8b218-kube-api-access-n54gf\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:54:18.059852 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:18.059823    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"
Oct 30 12:54:18.060021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:18.059847    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh" event={"ID":"c13ea519-9c63-4fb4-abc1-98f4eca8b218","Type":"ContainerDied","Data":"6bf5c65974df730df5897608f366c7f3b0b97f0399496714468d02edb337281b"}
Oct 30 12:54:18.060021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:18.059917    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bf5c65974df730df5897608f366c7f3b0b97f0399496714468d02edb337281b"
Oct 30 12:54:24.066548 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.066515    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"]
Oct 30 12:54:24.067147 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.067134    5837 cpu_manager.go:414] "containerMap: removing container" podUID="c13ea519-9c63-4fb4-abc1-98f4eca8b218" containerName="revoke-keys"
Oct 30 12:54:24.067189 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.067148    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13ea519-9c63-4fb4-abc1-98f4eca8b218" containerName="revoke-keys"
Oct 30 12:54:24.067303 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.067294    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13ea519-9c63-4fb4-abc1-98f4eca8b218" containerName="revoke-keys"
Oct 30 12:54:24.070708 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.070693    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"
Oct 30 12:54:24.087784 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.087759    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"]
Oct 30 12:54:24.095238 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.095212    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:54:24.095382 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.095304    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:54:24.096633 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.096614    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:54:24.096751 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.096673    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:54:24.171228 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.171195    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d339443-09e7-4990-8fcc-bfd0e6532dbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa897177-2j8bj\" (UID: \"0d339443-09e7-4990-8fcc-bfd0e6532dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"
Oct 30 12:54:24.171445 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.171389    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d339443-09e7-4990-8fcc-bfd0e6532dbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa897177-2j8bj\" (UID: \"0d339443-09e7-4990-8fcc-bfd0e6532dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"
Oct 30 12:54:24.171567 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.171553    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx75p\" (UniqueName: \"kubernetes.io/projected/0d339443-09e7-4990-8fcc-bfd0e6532dbe-kube-api-access-qx75p\") pod \"maas-api-revoke-sub-aa897177-2j8bj\" (UID: \"0d339443-09e7-4990-8fcc-bfd0e6532dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"
Oct 30 12:54:24.273284 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.273259    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx75p\" (UniqueName: \"kubernetes.io/projected/0d339443-09e7-4990-8fcc-bfd0e6532dbe-kube-api-access-qx75p\") pod \"maas-api-revoke-sub-aa897177-2j8bj\" (UID: \"0d339443-09e7-4990-8fcc-bfd0e6532dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"
Oct 30 12:54:24.273430 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.273362    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d339443-09e7-4990-8fcc-bfd0e6532dbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa897177-2j8bj\" (UID: \"0d339443-09e7-4990-8fcc-bfd0e6532dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"
Oct 30 12:54:24.273499 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.273485    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d339443-09e7-4990-8fcc-bfd0e6532dbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa897177-2j8bj\" (UID: \"0d339443-09e7-4990-8fcc-bfd0e6532dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"
Oct 30 12:54:24.274017 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.273996    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d339443-09e7-4990-8fcc-bfd0e6532dbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa897177-2j8bj\" (UID: \"0d339443-09e7-4990-8fcc-bfd0e6532dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"
Oct 30 12:54:24.275912 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.275892    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d339443-09e7-4990-8fcc-bfd0e6532dbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa897177-2j8bj\" (UID: \"0d339443-09e7-4990-8fcc-bfd0e6532dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"
Oct 30 12:54:24.305775 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.305734    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx75p\" (UniqueName: \"kubernetes.io/projected/0d339443-09e7-4990-8fcc-bfd0e6532dbe-kube-api-access-qx75p\") pod \"maas-api-revoke-sub-aa897177-2j8bj\" (UID: \"0d339443-09e7-4990-8fcc-bfd0e6532dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"
Oct 30 12:54:24.384961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.384873    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"
Oct 30 12:54:24.688756 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:24.688729    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"]
Oct 30 12:54:24.689658 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:54:24.689634    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d339443_09e7_4990_8fcc_bfd0e6532dbe.slice/crio-ed31d143cbc1e8de6bbeffd5874532e65a334b5bd408a13f3a41fe5df61861e7 WatchSource:0}: Error finding container ed31d143cbc1e8de6bbeffd5874532e65a334b5bd408a13f3a41fe5df61861e7: Status 404 returned error can't find the container with id ed31d143cbc1e8de6bbeffd5874532e65a334b5bd408a13f3a41fe5df61861e7
Oct 30 12:54:25.133187 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:25.133146    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj" event={"ID":"0d339443-09e7-4990-8fcc-bfd0e6532dbe","Type":"ContainerStarted","Data":"d2995074f1b84bca3027258868bef0165534dd4c4f8e22a2536acade2bbeb3ac"}
Oct 30 12:54:25.133572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:25.133213    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj" event={"ID":"0d339443-09e7-4990-8fcc-bfd0e6532dbe","Type":"ContainerStarted","Data":"ed31d143cbc1e8de6bbeffd5874532e65a334b5bd408a13f3a41fe5df61861e7"}
Oct 30 12:54:25.172788 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:25.172744    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj" podStartSLOduration=1.172731784 podStartE2EDuration="1.172731784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:25.169778333 +0000 UTC m=+1053.694062553" watchObservedRunningTime="2026-10-30 12:54:25.172731784 +0000 UTC m=+1053.697016003"
Oct 30 12:54:26.142116 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:26.142080    5837 generic.go:356] "Generic (PLEG): container finished" podID="0d339443-09e7-4990-8fcc-bfd0e6532dbe" containerID="d2995074f1b84bca3027258868bef0165534dd4c4f8e22a2536acade2bbeb3ac" exitCode=0
Oct 30 12:54:26.142554 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:26.142170    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj" event={"ID":"0d339443-09e7-4990-8fcc-bfd0e6532dbe","Type":"ContainerDied","Data":"d2995074f1b84bca3027258868bef0165534dd4c4f8e22a2536acade2bbeb3ac"}
Oct 30 12:54:27.322789 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:27.322758    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"
Oct 30 12:54:27.419110 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:27.419040    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d339443-09e7-4990-8fcc-bfd0e6532dbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d339443-09e7-4990-8fcc-bfd0e6532dbe-maas-api-xks-ca\") pod \"0d339443-09e7-4990-8fcc-bfd0e6532dbe\" (UID: \"0d339443-09e7-4990-8fcc-bfd0e6532dbe\") "
Oct 30 12:54:27.419110 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:27.419075    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d339443-09e7-4990-8fcc-bfd0e6532dbe-kube-api-access-qx75p\" (UniqueName: \"kubernetes.io/projected/0d339443-09e7-4990-8fcc-bfd0e6532dbe-kube-api-access-qx75p\") pod \"0d339443-09e7-4990-8fcc-bfd0e6532dbe\" (UID: \"0d339443-09e7-4990-8fcc-bfd0e6532dbe\") "
Oct 30 12:54:27.419318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:27.419213    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d339443-09e7-4990-8fcc-bfd0e6532dbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d339443-09e7-4990-8fcc-bfd0e6532dbe-maas-api-openshift-ca\") pod \"0d339443-09e7-4990-8fcc-bfd0e6532dbe\" (UID: \"0d339443-09e7-4990-8fcc-bfd0e6532dbe\") "
Oct 30 12:54:27.419621 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:27.419598    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d339443-09e7-4990-8fcc-bfd0e6532dbe-maas-api-openshift-ca" pod "0d339443-09e7-4990-8fcc-bfd0e6532dbe" (UID: "0d339443-09e7-4990-8fcc-bfd0e6532dbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:54:27.419840 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:27.419823    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d339443-09e7-4990-8fcc-bfd0e6532dbe-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:54:27.421489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:27.421453    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d339443-09e7-4990-8fcc-bfd0e6532dbe-maas-api-xks-ca" pod "0d339443-09e7-4990-8fcc-bfd0e6532dbe" (UID: "0d339443-09e7-4990-8fcc-bfd0e6532dbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:54:27.421596 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:27.421554    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d339443-09e7-4990-8fcc-bfd0e6532dbe-kube-api-access-qx75p" pod "0d339443-09e7-4990-8fcc-bfd0e6532dbe" (UID: "0d339443-09e7-4990-8fcc-bfd0e6532dbe"). InnerVolumeSpecName "kube-api-access-qx75p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:54:27.521142 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:27.521111    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d339443-09e7-4990-8fcc-bfd0e6532dbe-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:54:27.521142 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:27.521142    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx75p\" (UniqueName: \"kubernetes.io/projected/0d339443-09e7-4990-8fcc-bfd0e6532dbe-kube-api-access-qx75p\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:54:28.158084 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:28.158054    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"
Oct 30 12:54:28.158240 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:28.158080    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj" event={"ID":"0d339443-09e7-4990-8fcc-bfd0e6532dbe","Type":"ContainerDied","Data":"ed31d143cbc1e8de6bbeffd5874532e65a334b5bd408a13f3a41fe5df61861e7"}
Oct 30 12:54:28.158240 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:28.158124    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed31d143cbc1e8de6bbeffd5874532e65a334b5bd408a13f3a41fe5df61861e7"
Oct 30 12:54:29.055278 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:29.055230    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"]
Oct 30 12:54:29.063934 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:29.063869    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc34e4d-cpzbg"]
Oct 30 12:54:29.606100 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:29.606067    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ff0f3d-11af-46b6-a565-187e8b7207e1" path="/var/lib/kubelet/pods/b4ff0f3d-11af-46b6-a565-187e8b7207e1/volumes"
Oct 30 12:54:36.032882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:36.032829    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"]
Oct 30 12:54:36.040155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:36.040129    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0005f0-7kqml"]
Oct 30 12:54:37.132816 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.132783    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"]
Oct 30 12:54:37.133518 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.133502    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0d339443-09e7-4990-8fcc-bfd0e6532dbe" containerName="revoke-keys"
Oct 30 12:54:37.133582 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.133519    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d339443-09e7-4990-8fcc-bfd0e6532dbe" containerName="revoke-keys"
Oct 30 12:54:37.133650 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.133635    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d339443-09e7-4990-8fcc-bfd0e6532dbe" containerName="revoke-keys"
Oct 30 12:54:37.137840 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.137825    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"
Oct 30 12:54:37.155332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.155305    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"]
Oct 30 12:54:37.162113 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.162072    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:54:37.162113 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.162104    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:54:37.163627 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.163586    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:54:37.163796 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.163778    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:54:37.288007 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.287980    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e47c1373-llcp5\" (UID: \"b7e0add5-e41c-43c6-9557-f21f19e4ff3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"
Oct 30 12:54:37.288208 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.288186    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e47c1373-llcp5\" (UID: \"b7e0add5-e41c-43c6-9557-f21f19e4ff3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"
Oct 30 12:54:37.288291 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.288277    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgpr2\" (UniqueName: \"kubernetes.io/projected/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-kube-api-access-hgpr2\") pod \"maas-api-revoke-sub-e47c1373-llcp5\" (UID: \"b7e0add5-e41c-43c6-9557-f21f19e4ff3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"
Oct 30 12:54:37.389697 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.389622    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e47c1373-llcp5\" (UID: \"b7e0add5-e41c-43c6-9557-f21f19e4ff3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"
Oct 30 12:54:37.389843 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.389733    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e47c1373-llcp5\" (UID: \"b7e0add5-e41c-43c6-9557-f21f19e4ff3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"
Oct 30 12:54:37.389843 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.389768    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgpr2\" (UniqueName: \"kubernetes.io/projected/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-kube-api-access-hgpr2\") pod \"maas-api-revoke-sub-e47c1373-llcp5\" (UID: \"b7e0add5-e41c-43c6-9557-f21f19e4ff3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"
Oct 30 12:54:37.390450 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.390434    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e47c1373-llcp5\" (UID: \"b7e0add5-e41c-43c6-9557-f21f19e4ff3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"
Oct 30 12:54:37.392090 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.392062    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e47c1373-llcp5\" (UID: \"b7e0add5-e41c-43c6-9557-f21f19e4ff3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"
Oct 30 12:54:37.408613 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.408593    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgpr2\" (UniqueName: \"kubernetes.io/projected/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-kube-api-access-hgpr2\") pod \"maas-api-revoke-sub-e47c1373-llcp5\" (UID: \"b7e0add5-e41c-43c6-9557-f21f19e4ff3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"
Oct 30 12:54:37.452705 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.452685    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"
Oct 30 12:54:37.607644 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.607614    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d7ffb87-9c58-44f7-a83a-ef0d6866e951" path="/var/lib/kubelet/pods/2d7ffb87-9c58-44f7-a83a-ef0d6866e951/volumes"
Oct 30 12:54:37.764978 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:37.764937    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"]
Oct 30 12:54:38.237548 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:38.237509    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5" event={"ID":"b7e0add5-e41c-43c6-9557-f21f19e4ff3a","Type":"ContainerStarted","Data":"267f955ce4c8140fb04af0dbd345193c3d0ec40d4875e8533370cd41ceca82c6"}
Oct 30 12:54:38.238201 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:38.237575    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5" event={"ID":"b7e0add5-e41c-43c6-9557-f21f19e4ff3a","Type":"ContainerStarted","Data":"6226b6bb04b72bd866434702403463e25f63eb6447c0b38129848d19bcc8400d"}
Oct 30 12:54:38.277283 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:38.277233    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5" podStartSLOduration=1.277220029 podStartE2EDuration="1.277220029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:38.27473073 +0000 UTC m=+1066.799014949" watchObservedRunningTime="2026-10-30 12:54:38.277220029 +0000 UTC m=+1066.801504246"
Oct 30 12:54:39.028628 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:39.028598    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"]
Oct 30 12:54:39.039469 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:39.039446    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3caae70-bwjqt"]
Oct 30 12:54:39.251451 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:39.251371    5837 generic.go:356] "Generic (PLEG): container finished" podID="b7e0add5-e41c-43c6-9557-f21f19e4ff3a" containerID="267f955ce4c8140fb04af0dbd345193c3d0ec40d4875e8533370cd41ceca82c6" exitCode=0
Oct 30 12:54:39.251956 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:39.251466    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5" event={"ID":"b7e0add5-e41c-43c6-9557-f21f19e4ff3a","Type":"ContainerDied","Data":"267f955ce4c8140fb04af0dbd345193c3d0ec40d4875e8533370cd41ceca82c6"}
Oct 30 12:54:39.606471 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:39.606443    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af17d7f-a451-48d7-a03d-c25693a988f7" path="/var/lib/kubelet/pods/6af17d7f-a451-48d7-a03d-c25693a988f7/volumes"
Oct 30 12:54:40.431309 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:40.431284    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"
Oct 30 12:54:40.483318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:40.483287    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-kube-api-access-hgpr2\" (UniqueName: \"kubernetes.io/projected/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-kube-api-access-hgpr2\") pod \"b7e0add5-e41c-43c6-9557-f21f19e4ff3a\" (UID: \"b7e0add5-e41c-43c6-9557-f21f19e4ff3a\") "
Oct 30 12:54:40.483561 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:40.483358    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-maas-api-xks-ca\") pod \"b7e0add5-e41c-43c6-9557-f21f19e4ff3a\" (UID: \"b7e0add5-e41c-43c6-9557-f21f19e4ff3a\") "
Oct 30 12:54:40.483561 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:40.483470    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-maas-api-openshift-ca\") pod \"b7e0add5-e41c-43c6-9557-f21f19e4ff3a\" (UID: \"b7e0add5-e41c-43c6-9557-f21f19e4ff3a\") "
Oct 30 12:54:40.484364 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:40.484340    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-maas-api-openshift-ca" pod "b7e0add5-e41c-43c6-9557-f21f19e4ff3a" (UID: "b7e0add5-e41c-43c6-9557-f21f19e4ff3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:54:40.486802 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:40.486774    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-maas-api-xks-ca" pod "b7e0add5-e41c-43c6-9557-f21f19e4ff3a" (UID: "b7e0add5-e41c-43c6-9557-f21f19e4ff3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:54:40.493718 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:40.493694    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-kube-api-access-hgpr2" pod "b7e0add5-e41c-43c6-9557-f21f19e4ff3a" (UID: "b7e0add5-e41c-43c6-9557-f21f19e4ff3a"). InnerVolumeSpecName "kube-api-access-hgpr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:54:40.585806 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:40.585732    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgpr2\" (UniqueName: \"kubernetes.io/projected/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-kube-api-access-hgpr2\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:54:40.585806 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:40.585759    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:54:40.585806 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:40.585782    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e0add5-e41c-43c6-9557-f21f19e4ff3a-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:54:41.269823 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:41.269800    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"
Oct 30 12:54:41.270007 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:41.269835    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5" event={"ID":"b7e0add5-e41c-43c6-9557-f21f19e4ff3a","Type":"ContainerDied","Data":"6226b6bb04b72bd866434702403463e25f63eb6447c0b38129848d19bcc8400d"}
Oct 30 12:54:41.270007 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:41.269890    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6226b6bb04b72bd866434702403463e25f63eb6447c0b38129848d19bcc8400d"
Oct 30 12:54:45.097197 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:54:45.097155    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:54:45.574555 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.574522    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"]
Oct 30 12:54:45.575272 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.575249    5837 cpu_manager.go:414] "containerMap: removing container" podUID="b7e0add5-e41c-43c6-9557-f21f19e4ff3a" containerName="revoke-keys"
Oct 30 12:54:45.575272 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.575269    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7e0add5-e41c-43c6-9557-f21f19e4ff3a" containerName="revoke-keys"
Oct 30 12:54:45.575482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.575420    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7e0add5-e41c-43c6-9557-f21f19e4ff3a" containerName="revoke-keys"
Oct 30 12:54:45.579624 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.579604    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"
Oct 30 12:54:45.596639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.596621    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f9e9fd\""
Oct 30 12:54:45.613137 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.613109    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"]
Oct 30 12:54:45.650758 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.650727    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"]
Oct 30 12:54:45.655259 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.655240    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"
Oct 30 12:54:45.670719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.670681    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq\" (UID: \"c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"
Oct 30 12:54:45.670875 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.670852    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42t96\" (UniqueName: \"kubernetes.io/projected/c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed-kube-api-access-42t96\") pod \"payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq\" (UID: \"c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"
Oct 30 12:54:45.673825 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.673800    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"]
Oct 30 12:54:45.748420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.737713    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f9e9fd-dockercfg-r5m7n\""
Oct 30 12:54:45.782475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.780301    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42t96\" (UniqueName: \"kubernetes.io/projected/c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed-kube-api-access-42t96\") pod \"payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq\" (UID: \"c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"
Oct 30 12:54:45.782475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.780349    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8gl5\" (UniqueName: \"kubernetes.io/projected/67ad4195-3d80-4201-b3b9-fab610272e24-kube-api-access-n8gl5\") pod \"payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf\" (UID: \"67ad4195-3d80-4201-b3b9-fab610272e24\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"
Oct 30 12:54:45.782475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.780482    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ad4195-3d80-4201-b3b9-fab610272e24-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf\" (UID: \"67ad4195-3d80-4201-b3b9-fab610272e24\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"
Oct 30 12:54:45.782475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.780785    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq\" (UID: \"c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"
Oct 30 12:54:45.782864 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.782837    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq\" (UID: \"c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"
Oct 30 12:54:45.852296 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.852214    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42t96\" (UniqueName: \"kubernetes.io/projected/c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed-kube-api-access-42t96\") pod \"payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq\" (UID: \"c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"
Oct 30 12:54:45.900214 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.889528    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8gl5\" (UniqueName: \"kubernetes.io/projected/67ad4195-3d80-4201-b3b9-fab610272e24-kube-api-access-n8gl5\") pod \"payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf\" (UID: \"67ad4195-3d80-4201-b3b9-fab610272e24\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"
Oct 30 12:54:45.900214 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.889589    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ad4195-3d80-4201-b3b9-fab610272e24-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf\" (UID: \"67ad4195-3d80-4201-b3b9-fab610272e24\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"
Oct 30 12:54:45.900214 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.899608    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ad4195-3d80-4201-b3b9-fab610272e24-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf\" (UID: \"67ad4195-3d80-4201-b3b9-fab610272e24\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"
Oct 30 12:54:45.900214 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.899910    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"
Oct 30 12:54:45.919879 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.919845    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8gl5\" (UniqueName: \"kubernetes.io/projected/67ad4195-3d80-4201-b3b9-fab610272e24-kube-api-access-n8gl5\") pod \"payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf\" (UID: \"67ad4195-3d80-4201-b3b9-fab610272e24\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"
Oct 30 12:54:45.968810 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:45.968522    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"
Oct 30 12:54:46.338843 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:46.338600    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"]
Oct 30 12:54:46.383786 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:46.383752    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"]
Oct 30 12:54:46.388137 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:46.387994    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq" event={"ID":"c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed","Type":"ContainerStarted","Data":"7967a7d962be42653c9be2f866402dfeea9383ad12751e0e3a886af5f38117ea"}
Oct 30 12:54:46.409748 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:54:46.409713    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ad4195_3d80_4201_b3b9_fab610272e24.slice/crio-da8069d5125be695cfa66f1a444fb66f6f486ab6fe22c2e30318c478e1231cb4 WatchSource:0}: Error finding container da8069d5125be695cfa66f1a444fb66f6f486ab6fe22c2e30318c478e1231cb4: Status 404 returned error can't find the container with id da8069d5125be695cfa66f1a444fb66f6f486ab6fe22c2e30318c478e1231cb4
Oct 30 12:54:47.433064 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:47.433005    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf" event={"ID":"67ad4195-3d80-4201-b3b9-fab610272e24","Type":"ContainerStarted","Data":"fb1e548b13fa5d6bdcd66018944a0915bcc141551039b7b5a49414f1f9b869d5"}
Oct 30 12:54:47.435023 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:47.434980    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf" event={"ID":"67ad4195-3d80-4201-b3b9-fab610272e24","Type":"ContainerStarted","Data":"da8069d5125be695cfa66f1a444fb66f6f486ab6fe22c2e30318c478e1231cb4"}
Oct 30 12:54:47.435257 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:47.435244    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"
Oct 30 12:54:47.437861 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:47.437837    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq" event={"ID":"c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed","Type":"ContainerStarted","Data":"bc988f870ff9c26a2f5f249af80f146db0845d94289770ed868a8755e0b3de31"}
Oct 30 12:54:47.448787 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:47.448756    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"
Oct 30 12:54:47.511204 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:47.511127    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf" podStartSLOduration=2.511103769 podStartE2EDuration="2.511103769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:47.50820763 +0000 UTC m=+1076.032491850" watchObservedRunningTime="2026-10-30 12:54:47.511103769 +0000 UTC m=+1076.035387989"
Oct 30 12:54:47.560698 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:47.560639    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq" podStartSLOduration=2.560619305 podStartE2EDuration="2.560619305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:47.556852201 +0000 UTC m=+1076.081136420" watchObservedRunningTime="2026-10-30 12:54:47.560619305 +0000 UTC m=+1076.084903525"
Oct 30 12:54:49.450424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:49.449495    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:49.577421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:49.577249    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:49.880766 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:49.880722    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"]
Oct 30 12:54:50.002227 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.002197    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.070350 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.070299    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def19b51-148c-4a5e-bdbe-313b17e37303-istio-podinfo\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.070589 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.070426    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-istio-envoy\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.070589 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.070519    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-workload-socket\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.070589 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.070580    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def19b51-148c-4a5e-bdbe-313b17e37303-istio-token\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.070765 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.070641    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcl77\" (UniqueName: \"kubernetes.io/projected/def19b51-148c-4a5e-bdbe-313b17e37303-kube-api-access-mcl77\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.070765 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.070689    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def19b51-148c-4a5e-bdbe-313b17e37303-istiod-ca-cert\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.070765 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.070749    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-workload-certs\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.070913 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.070778    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-credential-socket\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.070913 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.070820    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-istio-data\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.352420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.349909    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-workload-socket\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.352420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.349994    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def19b51-148c-4a5e-bdbe-313b17e37303-istio-token\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.352420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.350057    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcl77\" (UniqueName: \"kubernetes.io/projected/def19b51-148c-4a5e-bdbe-313b17e37303-kube-api-access-mcl77\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.352420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.350108    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def19b51-148c-4a5e-bdbe-313b17e37303-istiod-ca-cert\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.352420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.350164    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-workload-certs\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.352420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.350193    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-credential-socket\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.352420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.350235    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-istio-data\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.352420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.350413    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def19b51-148c-4a5e-bdbe-313b17e37303-istio-podinfo\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.352420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.350480    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-istio-envoy\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.377665 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.374288    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6bda55-openshift-default-dockercfg-7z8fg\""
Oct 30 12:54:50.377665 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.270379    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"]
Oct 30 12:54:50.377665 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.375916    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def19b51-148c-4a5e-bdbe-313b17e37303-istiod-ca-cert\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.394849 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.388747    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-credential-socket\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.394849 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.389036    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-workload-certs\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.394849 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.389871    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-istio-data\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.401838 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.401798    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-workload-socket\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.495443 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.453122    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:50.495443 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.454188    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:54:50.495443 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.470140    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def19b51-148c-4a5e-bdbe-313b17e37303-istio-token\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.495443 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.494848    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:54:50.534428 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.528832    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-istio-envoy\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.565435 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.555870    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def19b51-148c-4a5e-bdbe-313b17e37303-istio-podinfo\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.565435 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.564119    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcl77\" (UniqueName: \"kubernetes.io/projected/def19b51-148c-4a5e-bdbe-313b17e37303-kube-api-access-mcl77\") pod \"e2e-api-a-6bda55-openshift-default-757659f87-x82c7\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") " pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.831421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.814526    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:54:50.855422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.833684    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:50.953527 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:50.953483    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:54:51.513821 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:51.513770    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:52.285420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:52.276071    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:54:52.371674 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:52.371631    5837 scope.go:122] "RemoveContainer" containerID="94fcba83a55da8a9f3f38b017191e847049f184e29e20c1235df86a853371401"
Oct 30 12:54:52.420290 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:52.420246    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:54:52.670200 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:52.519267    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:52.755644 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:52.733526    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:54:52.755823 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:52.755677    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:52.926516 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:52.925118    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:54:53.343216 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:53.343188    5837 scope.go:122] "RemoveContainer" containerID="d968412018163c179d9cd3ae8cfa9ef695a00f271ddc57cad649c81392aca343"
Oct 30 12:54:53.505480 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:53.352291    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:54:53.740423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:53.571749    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:54:53.960337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:53.950090    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.007562 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:54.007502    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:54:54.247791 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:54.247743    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:54:54.405686 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:54.405640    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:54:54.405955 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:54.405931    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:54:54.410580 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:54.259375    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"]
Oct 30 12:54:54.423673 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:54.423616    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.523668 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:54.523610    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.983919 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:54.976592    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:55.086326 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:55.086114    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:54:55.211426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:55.200032    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7" event={"ID":"def19b51-148c-4a5e-bdbe-313b17e37303","Type":"ContainerStarted","Data":"3bcdc4616f8c1a6ec21f9a3645f001f182edef01bdb22b58f7a128bbc5d0aaf1"}
Oct 30 12:54:55.524104 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:55.524064    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:55.742207 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:55.742170    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:54:55.742509 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:55.742476    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" podUID="faf87439-5044-43cd-9b1a-200188299f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:55.937788 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:55.937754    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:54:55.938126 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:55.938091    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:55.978006 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:55.977889    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:56.091881 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:56.091833    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:54:56.529417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:56.529135    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:56.994597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:56.988493    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:54:57.096988 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:57.093113    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:57.557342 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:57.533180    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:57.994476 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:57.994199    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:58.143183 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:58.143141    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7" event={"ID":"def19b51-148c-4a5e-bdbe-313b17e37303","Type":"ContainerStarted","Data":"26613084cc223bc1a6f00fd2c431804fd022ed504917173b3be64ba42fa4a691"}
Oct 30 12:54:58.144764 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:58.144741    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:54:58.173882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:58.173180    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7" podStartSLOduration=9.173159122 podStartE2EDuration="9.173159122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:58.166289714 +0000 UTC m=+1086.690573934" watchObservedRunningTime="2026-10-30 12:54:58.173159122 +0000 UTC m=+1086.697443368"
Oct 30 12:54:58.345470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:58.344434    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:54:58.363123 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:58.362323    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:54:58.844159 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:58.844127    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:58.850111 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:58.850086    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:59.121975 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:59.121874    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:59.123107 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:59.123078    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:54:59.537453 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:59.537377    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"
Oct 30 12:54:59.538313 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:54:59.538292    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"
Oct 30 12:55:02.973175 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:02.972963    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"]
Oct 30 12:55:02.986204 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:02.986175    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"
Oct 30 12:55:02.995968 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:02.995942    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"]
Oct 30 12:55:03.003532 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:03.002906    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:55:03.018697 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:03.018661    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:55:03.018697 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:03.018672    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:55:03.018942 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:03.018710    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:55:03.029741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:03.029711    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjv8h\" (UniqueName: \"kubernetes.io/projected/b93a9a05-8a65-4ca1-a749-cf7e7c476240-kube-api-access-xjv8h\") pod \"maas-api-revoke-sub-604a4dfa-rf6js\" (UID: \"b93a9a05-8a65-4ca1-a749-cf7e7c476240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"
Oct 30 12:55:03.029929 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:03.029771    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93a9a05-8a65-4ca1-a749-cf7e7c476240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-604a4dfa-rf6js\" (UID: \"b93a9a05-8a65-4ca1-a749-cf7e7c476240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"
Oct 30 12:55:03.029929 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:03.029814    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93a9a05-8a65-4ca1-a749-cf7e7c476240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-604a4dfa-rf6js\" (UID: \"b93a9a05-8a65-4ca1-a749-cf7e7c476240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"
Oct 30 12:55:03.131246 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:03.131210    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjv8h\" (UniqueName: \"kubernetes.io/projected/b93a9a05-8a65-4ca1-a749-cf7e7c476240-kube-api-access-xjv8h\") pod \"maas-api-revoke-sub-604a4dfa-rf6js\" (UID: \"b93a9a05-8a65-4ca1-a749-cf7e7c476240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"
Oct 30 12:55:03.131502 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:03.131287    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93a9a05-8a65-4ca1-a749-cf7e7c476240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-604a4dfa-rf6js\" (UID: \"b93a9a05-8a65-4ca1-a749-cf7e7c476240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"
Oct 30 12:55:03.131502 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:03.131349    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93a9a05-8a65-4ca1-a749-cf7e7c476240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-604a4dfa-rf6js\" (UID: \"b93a9a05-8a65-4ca1-a749-cf7e7c476240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"
Oct 30 12:55:03.132288 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:03.132264    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93a9a05-8a65-4ca1-a749-cf7e7c476240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-604a4dfa-rf6js\" (UID: \"b93a9a05-8a65-4ca1-a749-cf7e7c476240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"
Oct 30 12:55:03.133981 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:03.133957    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93a9a05-8a65-4ca1-a749-cf7e7c476240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-604a4dfa-rf6js\" (UID: \"b93a9a05-8a65-4ca1-a749-cf7e7c476240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"
Oct 30 12:55:03.150822 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:03.150784    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjv8h\" (UniqueName: \"kubernetes.io/projected/b93a9a05-8a65-4ca1-a749-cf7e7c476240-kube-api-access-xjv8h\") pod \"maas-api-revoke-sub-604a4dfa-rf6js\" (UID: \"b93a9a05-8a65-4ca1-a749-cf7e7c476240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"
Oct 30 12:55:03.315335 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:03.315301    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"
Oct 30 12:55:03.649197 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:03.649169    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"]
Oct 30 12:55:03.649846 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:55:03.649821    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93a9a05_8a65_4ca1_a749_cf7e7c476240.slice/crio-f51b196eef62c4e50132acdbe1e7cc8d3f56c4ae479a7f87567d0fd5ffafa401 WatchSource:0}: Error finding container f51b196eef62c4e50132acdbe1e7cc8d3f56c4ae479a7f87567d0fd5ffafa401: Status 404 returned error can't find the container with id f51b196eef62c4e50132acdbe1e7cc8d3f56c4ae479a7f87567d0fd5ffafa401
Oct 30 12:55:04.183720 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:04.183629    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js" event={"ID":"b93a9a05-8a65-4ca1-a749-cf7e7c476240","Type":"ContainerStarted","Data":"c1c87ab10ae795db8e04c0d58fe696b25a9799838f7635bd742bb93b0cd483cd"}
Oct 30 12:55:04.183720 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:04.183679    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js" event={"ID":"b93a9a05-8a65-4ca1-a749-cf7e7c476240","Type":"ContainerStarted","Data":"f51b196eef62c4e50132acdbe1e7cc8d3f56c4ae479a7f87567d0fd5ffafa401"}
Oct 30 12:55:04.233617 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:04.233563    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js" podStartSLOduration=2.23354777 podStartE2EDuration="2.23354777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:04.229789 +0000 UTC m=+1092.754073221" watchObservedRunningTime="2026-10-30 12:55:04.23354777 +0000 UTC m=+1092.757831989"
Oct 30 12:55:04.861540 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:04.859527    5837 patch_prober.go:29] interesting pod/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:55:04.861540 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:04.859606    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:55:05.195278 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:05.195187    5837 generic.go:356] "Generic (PLEG): container finished" podID="b93a9a05-8a65-4ca1-a749-cf7e7c476240" containerID="c1c87ab10ae795db8e04c0d58fe696b25a9799838f7635bd742bb93b0cd483cd" exitCode=0
Oct 30 12:55:05.195278 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:05.195272    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js" event={"ID":"b93a9a05-8a65-4ca1-a749-cf7e7c476240","Type":"ContainerDied","Data":"c1c87ab10ae795db8e04c0d58fe696b25a9799838f7635bd742bb93b0cd483cd"}
Oct 30 12:55:06.205220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.205189    5837 generic.go:356] "Generic (PLEG): container finished" podID="484f0dd9-59d8-4690-a051-044c1fed4945" containerID="85593eac598b59dc3265222df1e10753f93785187b65e244934970880951cd99" exitCode=0
Oct 30 12:55:06.205725 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.205271    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" event={"ID":"484f0dd9-59d8-4690-a051-044c1fed4945","Type":"ContainerDied","Data":"85593eac598b59dc3265222df1e10753f93785187b65e244934970880951cd99"}
Oct 30 12:55:06.206155 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.205802    5837 scope.go:122] "RemoveContainer" containerID="85593eac598b59dc3265222df1e10753f93785187b65e244934970880951cd99"
Oct 30 12:55:06.207312 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.207280    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:55:06.207432 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.207346    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:55:06.207432 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.207416    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:55:06.491306 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.491265    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"
Oct 30 12:55:06.516234 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.514490    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b93a9a05-8a65-4ca1-a749-cf7e7c476240-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93a9a05-8a65-4ca1-a749-cf7e7c476240-maas-api-xks-ca\") pod \"b93a9a05-8a65-4ca1-a749-cf7e7c476240\" (UID: \"b93a9a05-8a65-4ca1-a749-cf7e7c476240\") "
Oct 30 12:55:06.516234 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.514626    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b93a9a05-8a65-4ca1-a749-cf7e7c476240-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93a9a05-8a65-4ca1-a749-cf7e7c476240-maas-api-openshift-ca\") pod \"b93a9a05-8a65-4ca1-a749-cf7e7c476240\" (UID: \"b93a9a05-8a65-4ca1-a749-cf7e7c476240\") "
Oct 30 12:55:06.517983 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.517858    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93a9a05-8a65-4ca1-a749-cf7e7c476240-maas-api-openshift-ca" pod "b93a9a05-8a65-4ca1-a749-cf7e7c476240" (UID: "b93a9a05-8a65-4ca1-a749-cf7e7c476240"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:06.518670 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.518630    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93a9a05-8a65-4ca1-a749-cf7e7c476240-kube-api-access-xjv8h\" (UniqueName: \"kubernetes.io/projected/b93a9a05-8a65-4ca1-a749-cf7e7c476240-kube-api-access-xjv8h\") pod \"b93a9a05-8a65-4ca1-a749-cf7e7c476240\" (UID: \"b93a9a05-8a65-4ca1-a749-cf7e7c476240\") "
Oct 30 12:55:06.520312 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.520267    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93a9a05-8a65-4ca1-a749-cf7e7c476240-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:06.522409 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.521687    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93a9a05-8a65-4ca1-a749-cf7e7c476240-maas-api-xks-ca" pod "b93a9a05-8a65-4ca1-a749-cf7e7c476240" (UID: "b93a9a05-8a65-4ca1-a749-cf7e7c476240"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:55:06.537715 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.537680    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93a9a05-8a65-4ca1-a749-cf7e7c476240-kube-api-access-xjv8h" pod "b93a9a05-8a65-4ca1-a749-cf7e7c476240" (UID: "b93a9a05-8a65-4ca1-a749-cf7e7c476240"). InnerVolumeSpecName "kube-api-access-xjv8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:06.622149 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.622119    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93a9a05-8a65-4ca1-a749-cf7e7c476240-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:06.622149 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:06.622152    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjv8h\" (UniqueName: \"kubernetes.io/projected/b93a9a05-8a65-4ca1-a749-cf7e7c476240-kube-api-access-xjv8h\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:07.243679 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:07.243648    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" event={"ID":"484f0dd9-59d8-4690-a051-044c1fed4945","Type":"ContainerStarted","Data":"7db390e27e9a0dce6541a6f06d607836e7c7f35686073bb3475bfdbc60106ad4"}
Oct 30 12:55:07.245665 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:07.245639    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"
Oct 30 12:55:07.245818 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:07.245788    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js" event={"ID":"b93a9a05-8a65-4ca1-a749-cf7e7c476240","Type":"ContainerDied","Data":"f51b196eef62c4e50132acdbe1e7cc8d3f56c4ae479a7f87567d0fd5ffafa401"}
Oct 30 12:55:07.245882 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:07.245834    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f51b196eef62c4e50132acdbe1e7cc8d3f56c4ae479a7f87567d0fd5ffafa401"
Oct 30 12:55:07.279163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:07.279128    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:55:07.279387 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:07.279365    5837 patch_prober.go:29] interesting pod/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 12:55:07.279554 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:07.279532    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 12:55:08.280127 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:08.280097    5837 patch_prober.go:29] interesting pod/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 12:55:08.280574 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:08.280156    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 12:55:09.279814 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:09.279775    5837 patch_prober.go:29] interesting pod/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 12:55:09.280011 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:09.279829    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 12:55:10.280029 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:10.279992    5837 patch_prober.go:29] interesting pod/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 12:55:10.280437 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:10.280054    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 12:55:11.279484 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:11.279451    5837 patch_prober.go:29] interesting pod/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 12:55:11.279666 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:11.279519    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 12:55:12.279543 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:12.279503    5837 patch_prober.go:29] interesting pod/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 12:55:12.279997 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:12.279579    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 12:55:13.283773 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:13.283746    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:55:13.284288 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:13.284270    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:55:13.285035 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:13.285009    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:55:15.096076 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:55:15.096038    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:55:16.409518 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.409487    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"]
Oct 30 12:55:16.411651 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.411622    5837 cpu_manager.go:414] "containerMap: removing container" podUID="b93a9a05-8a65-4ca1-a749-cf7e7c476240" containerName="revoke-keys"
Oct 30 12:55:16.411824 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.411812    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93a9a05-8a65-4ca1-a749-cf7e7c476240" containerName="revoke-keys"
Oct 30 12:55:16.412242 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.412224    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93a9a05-8a65-4ca1-a749-cf7e7c476240" containerName="revoke-keys"
Oct 30 12:55:16.417429 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.417408    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"
Oct 30 12:55:16.427051 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.427029    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"]
Oct 30 12:55:16.432569 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.432542    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:55:16.432569 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.432542    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:55:16.432720 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.432556    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:55:16.432720 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.432633    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:55:16.553798 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.553763    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7244a952-78d0-4156-ab8b-95bb89a4617b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4beb2220-hv6bl\" (UID: \"7244a952-78d0-4156-ab8b-95bb89a4617b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"
Oct 30 12:55:16.554000 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.553894    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7244a952-78d0-4156-ab8b-95bb89a4617b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4beb2220-hv6bl\" (UID: \"7244a952-78d0-4156-ab8b-95bb89a4617b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"
Oct 30 12:55:16.554063 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.554026    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmwtn\" (UniqueName: \"kubernetes.io/projected/7244a952-78d0-4156-ab8b-95bb89a4617b-kube-api-access-tmwtn\") pod \"maas-api-revoke-sub-4beb2220-hv6bl\" (UID: \"7244a952-78d0-4156-ab8b-95bb89a4617b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"
Oct 30 12:55:16.655769 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.655732    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7244a952-78d0-4156-ab8b-95bb89a4617b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4beb2220-hv6bl\" (UID: \"7244a952-78d0-4156-ab8b-95bb89a4617b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"
Oct 30 12:55:16.655964 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.655844    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7244a952-78d0-4156-ab8b-95bb89a4617b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4beb2220-hv6bl\" (UID: \"7244a952-78d0-4156-ab8b-95bb89a4617b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"
Oct 30 12:55:16.655964 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.655947    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmwtn\" (UniqueName: \"kubernetes.io/projected/7244a952-78d0-4156-ab8b-95bb89a4617b-kube-api-access-tmwtn\") pod \"maas-api-revoke-sub-4beb2220-hv6bl\" (UID: \"7244a952-78d0-4156-ab8b-95bb89a4617b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"
Oct 30 12:55:16.656535 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.656513    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7244a952-78d0-4156-ab8b-95bb89a4617b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4beb2220-hv6bl\" (UID: \"7244a952-78d0-4156-ab8b-95bb89a4617b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"
Oct 30 12:55:16.658281 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.658263    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7244a952-78d0-4156-ab8b-95bb89a4617b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4beb2220-hv6bl\" (UID: \"7244a952-78d0-4156-ab8b-95bb89a4617b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"
Oct 30 12:55:16.674747 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.674691    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmwtn\" (UniqueName: \"kubernetes.io/projected/7244a952-78d0-4156-ab8b-95bb89a4617b-kube-api-access-tmwtn\") pod \"maas-api-revoke-sub-4beb2220-hv6bl\" (UID: \"7244a952-78d0-4156-ab8b-95bb89a4617b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"
Oct 30 12:55:16.734602 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:16.734569    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"
Oct 30 12:55:17.039610 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:55:17.039581    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7244a952_78d0_4156_ab8b_95bb89a4617b.slice/crio-8ce9acac35b4553fa15955f666b6adf81ba0d098a875cfce27fcdc77033418c1 WatchSource:0}: Error finding container 8ce9acac35b4553fa15955f666b6adf81ba0d098a875cfce27fcdc77033418c1: Status 404 returned error can't find the container with id 8ce9acac35b4553fa15955f666b6adf81ba0d098a875cfce27fcdc77033418c1
Oct 30 12:55:17.039872 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:17.039851    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"]
Oct 30 12:55:17.363178 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:17.363149    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl" event={"ID":"7244a952-78d0-4156-ab8b-95bb89a4617b","Type":"ContainerStarted","Data":"0c522777a614bccb3873f46b7c2334527f76bbf19b4c3ab69572050d305941b5"}
Oct 30 12:55:17.363348 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:17.363197    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl" event={"ID":"7244a952-78d0-4156-ab8b-95bb89a4617b","Type":"ContainerStarted","Data":"8ce9acac35b4553fa15955f666b6adf81ba0d098a875cfce27fcdc77033418c1"}
Oct 30 12:55:17.422894 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:17.422828    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl" podStartSLOduration=1.422806139 podStartE2EDuration="1.422806139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:17.404376199 +0000 UTC m=+1105.928660421" watchObservedRunningTime="2026-10-30 12:55:17.422806139 +0000 UTC m=+1105.947090359"
Oct 30 12:55:18.385264 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:18.385231    5837 generic.go:356] "Generic (PLEG): container finished" podID="7244a952-78d0-4156-ab8b-95bb89a4617b" containerID="0c522777a614bccb3873f46b7c2334527f76bbf19b4c3ab69572050d305941b5" exitCode=0
Oct 30 12:55:18.385521 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:18.385315    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl" event={"ID":"7244a952-78d0-4156-ab8b-95bb89a4617b","Type":"ContainerDied","Data":"0c522777a614bccb3873f46b7c2334527f76bbf19b4c3ab69572050d305941b5"}
Oct 30 12:55:19.547542 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:19.547501    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"
Oct 30 12:55:19.596924 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:19.596891    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7244a952-78d0-4156-ab8b-95bb89a4617b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7244a952-78d0-4156-ab8b-95bb89a4617b-maas-api-openshift-ca\") pod \"7244a952-78d0-4156-ab8b-95bb89a4617b\" (UID: \"7244a952-78d0-4156-ab8b-95bb89a4617b\") "
Oct 30 12:55:19.597097 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:19.597044    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7244a952-78d0-4156-ab8b-95bb89a4617b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7244a952-78d0-4156-ab8b-95bb89a4617b-maas-api-xks-ca\") pod \"7244a952-78d0-4156-ab8b-95bb89a4617b\" (UID: \"7244a952-78d0-4156-ab8b-95bb89a4617b\") "
Oct 30 12:55:19.597186 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:19.597167    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7244a952-78d0-4156-ab8b-95bb89a4617b-kube-api-access-tmwtn\" (UniqueName: \"kubernetes.io/projected/7244a952-78d0-4156-ab8b-95bb89a4617b-kube-api-access-tmwtn\") pod \"7244a952-78d0-4156-ab8b-95bb89a4617b\" (UID: \"7244a952-78d0-4156-ab8b-95bb89a4617b\") "
Oct 30 12:55:19.597274 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:19.597248    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7244a952-78d0-4156-ab8b-95bb89a4617b-maas-api-openshift-ca" pod "7244a952-78d0-4156-ab8b-95bb89a4617b" (UID: "7244a952-78d0-4156-ab8b-95bb89a4617b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:19.597913 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:19.597840    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7244a952-78d0-4156-ab8b-95bb89a4617b-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:19.599556 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:19.599523    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7244a952-78d0-4156-ab8b-95bb89a4617b-kube-api-access-tmwtn" pod "7244a952-78d0-4156-ab8b-95bb89a4617b" (UID: "7244a952-78d0-4156-ab8b-95bb89a4617b"). InnerVolumeSpecName "kube-api-access-tmwtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:19.599746 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:19.599726    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7244a952-78d0-4156-ab8b-95bb89a4617b-maas-api-xks-ca" pod "7244a952-78d0-4156-ab8b-95bb89a4617b" (UID: "7244a952-78d0-4156-ab8b-95bb89a4617b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:55:19.699729 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:19.699661    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7244a952-78d0-4156-ab8b-95bb89a4617b-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:19.699729 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:19.699687    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmwtn\" (UniqueName: \"kubernetes.io/projected/7244a952-78d0-4156-ab8b-95bb89a4617b-kube-api-access-tmwtn\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.401723 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:20.401692    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"
Oct 30 12:55:20.401906 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:20.401727    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl" event={"ID":"7244a952-78d0-4156-ab8b-95bb89a4617b","Type":"ContainerDied","Data":"8ce9acac35b4553fa15955f666b6adf81ba0d098a875cfce27fcdc77033418c1"}
Oct 30 12:55:20.401906 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:20.401777    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ce9acac35b4553fa15955f666b6adf81ba0d098a875cfce27fcdc77033418c1"
Oct 30 12:55:26.811844 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:26.811809    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"]
Oct 30 12:55:26.812490 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:26.812476    5837 cpu_manager.go:414] "containerMap: removing container" podUID="7244a952-78d0-4156-ab8b-95bb89a4617b" containerName="revoke-keys"
Oct 30 12:55:26.812538 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:26.812491    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7244a952-78d0-4156-ab8b-95bb89a4617b" containerName="revoke-keys"
Oct 30 12:55:26.812630 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:26.812622    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="7244a952-78d0-4156-ab8b-95bb89a4617b" containerName="revoke-keys"
Oct 30 12:55:26.816368 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:26.816349    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"
Oct 30 12:55:26.834258 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:26.834234    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"]
Oct 30 12:55:26.840213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:26.840159    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:55:26.840330 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:26.840237    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:55:26.841810 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:26.841788    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:55:26.841923 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:26.841839    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:55:26.916375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:26.916345    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpmbf\" (UniqueName: \"kubernetes.io/projected/747e76a9-41d7-4942-a33f-53aa5fd53507-kube-api-access-cpmbf\") pod \"maas-api-revoke-sub-7d2ef82b-g8qxj\" (UID: \"747e76a9-41d7-4942-a33f-53aa5fd53507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"
Oct 30 12:55:26.916639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:26.916622    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/747e76a9-41d7-4942-a33f-53aa5fd53507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d2ef82b-g8qxj\" (UID: \"747e76a9-41d7-4942-a33f-53aa5fd53507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"
Oct 30 12:55:26.916682 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:26.916675    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/747e76a9-41d7-4942-a33f-53aa5fd53507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d2ef82b-g8qxj\" (UID: \"747e76a9-41d7-4942-a33f-53aa5fd53507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"
Oct 30 12:55:27.018246 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:27.018221    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/747e76a9-41d7-4942-a33f-53aa5fd53507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d2ef82b-g8qxj\" (UID: \"747e76a9-41d7-4942-a33f-53aa5fd53507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"
Oct 30 12:55:27.018364 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:27.018267    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpmbf\" (UniqueName: \"kubernetes.io/projected/747e76a9-41d7-4942-a33f-53aa5fd53507-kube-api-access-cpmbf\") pod \"maas-api-revoke-sub-7d2ef82b-g8qxj\" (UID: \"747e76a9-41d7-4942-a33f-53aa5fd53507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"
Oct 30 12:55:27.018456 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:27.018445    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/747e76a9-41d7-4942-a33f-53aa5fd53507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d2ef82b-g8qxj\" (UID: \"747e76a9-41d7-4942-a33f-53aa5fd53507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"
Oct 30 12:55:27.019011 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:27.018990    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/747e76a9-41d7-4942-a33f-53aa5fd53507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d2ef82b-g8qxj\" (UID: \"747e76a9-41d7-4942-a33f-53aa5fd53507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"
Oct 30 12:55:27.020757 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:27.020740    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/747e76a9-41d7-4942-a33f-53aa5fd53507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d2ef82b-g8qxj\" (UID: \"747e76a9-41d7-4942-a33f-53aa5fd53507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"
Oct 30 12:55:27.037442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:27.037380    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpmbf\" (UniqueName: \"kubernetes.io/projected/747e76a9-41d7-4942-a33f-53aa5fd53507-kube-api-access-cpmbf\") pod \"maas-api-revoke-sub-7d2ef82b-g8qxj\" (UID: \"747e76a9-41d7-4942-a33f-53aa5fd53507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"
Oct 30 12:55:27.130141 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:27.130063    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"
Oct 30 12:55:27.430243 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:55:27.430219    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747e76a9_41d7_4942_a33f_53aa5fd53507.slice/crio-493dedbc3c8d1fb35fb36420a758d13a375ea3f3ff78862f3a0fd02522a8a7eb WatchSource:0}: Error finding container 493dedbc3c8d1fb35fb36420a758d13a375ea3f3ff78862f3a0fd02522a8a7eb: Status 404 returned error can't find the container with id 493dedbc3c8d1fb35fb36420a758d13a375ea3f3ff78862f3a0fd02522a8a7eb
Oct 30 12:55:27.431485 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:27.431459    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"]
Oct 30 12:55:27.467319 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:27.467295    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj" event={"ID":"747e76a9-41d7-4942-a33f-53aa5fd53507","Type":"ContainerStarted","Data":"493dedbc3c8d1fb35fb36420a758d13a375ea3f3ff78862f3a0fd02522a8a7eb"}
Oct 30 12:55:28.476703 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:28.476620    5837 generic.go:356] "Generic (PLEG): container finished" podID="747e76a9-41d7-4942-a33f-53aa5fd53507" containerID="897298e09ac3e463dcac2a8a7e3804fbef37647db7dba74f37e39856d29fc5ac" exitCode=0
Oct 30 12:55:28.477054 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:28.476712    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj" event={"ID":"747e76a9-41d7-4942-a33f-53aa5fd53507","Type":"ContainerDied","Data":"897298e09ac3e463dcac2a8a7e3804fbef37647db7dba74f37e39856d29fc5ac"}
Oct 30 12:55:29.671856 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:29.671836    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"
Oct 30 12:55:29.764609 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:29.764522    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747e76a9-41d7-4942-a33f-53aa5fd53507-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/747e76a9-41d7-4942-a33f-53aa5fd53507-maas-api-openshift-ca\") pod \"747e76a9-41d7-4942-a33f-53aa5fd53507\" (UID: \"747e76a9-41d7-4942-a33f-53aa5fd53507\") "
Oct 30 12:55:29.764783 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:29.764643    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747e76a9-41d7-4942-a33f-53aa5fd53507-kube-api-access-cpmbf\" (UniqueName: \"kubernetes.io/projected/747e76a9-41d7-4942-a33f-53aa5fd53507-kube-api-access-cpmbf\") pod \"747e76a9-41d7-4942-a33f-53aa5fd53507\" (UID: \"747e76a9-41d7-4942-a33f-53aa5fd53507\") "
Oct 30 12:55:29.764892 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:29.764874    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/747e76a9-41d7-4942-a33f-53aa5fd53507-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/747e76a9-41d7-4942-a33f-53aa5fd53507-maas-api-xks-ca\") pod \"747e76a9-41d7-4942-a33f-53aa5fd53507\" (UID: \"747e76a9-41d7-4942-a33f-53aa5fd53507\") "
Oct 30 12:55:29.773724 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:29.765980    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747e76a9-41d7-4942-a33f-53aa5fd53507-maas-api-openshift-ca" pod "747e76a9-41d7-4942-a33f-53aa5fd53507" (UID: "747e76a9-41d7-4942-a33f-53aa5fd53507"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:29.773724 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:29.767753    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/747e76a9-41d7-4942-a33f-53aa5fd53507-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:29.784328 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:29.784283    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/747e76a9-41d7-4942-a33f-53aa5fd53507-maas-api-xks-ca" pod "747e76a9-41d7-4942-a33f-53aa5fd53507" (UID: "747e76a9-41d7-4942-a33f-53aa5fd53507"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:55:29.786931 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:29.786897    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747e76a9-41d7-4942-a33f-53aa5fd53507-kube-api-access-cpmbf" pod "747e76a9-41d7-4942-a33f-53aa5fd53507" (UID: "747e76a9-41d7-4942-a33f-53aa5fd53507"). InnerVolumeSpecName "kube-api-access-cpmbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:29.868945 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:29.868912    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/747e76a9-41d7-4942-a33f-53aa5fd53507-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:29.868945 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:29.868944    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpmbf\" (UniqueName: \"kubernetes.io/projected/747e76a9-41d7-4942-a33f-53aa5fd53507-kube-api-access-cpmbf\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:30.202542 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:30.202465    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"]
Oct 30 12:55:30.202720 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:30.202696    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s" podUID="1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" containerName="istio-proxy" containerID="cri-o://ba556f50c847aa6ffee2143e395b5783af0d19246baae41e91c215feccc67399" gracePeriod=30
Oct 30 12:55:30.495599 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:30.495567    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj" event={"ID":"747e76a9-41d7-4942-a33f-53aa5fd53507","Type":"ContainerDied","Data":"493dedbc3c8d1fb35fb36420a758d13a375ea3f3ff78862f3a0fd02522a8a7eb"}
Oct 30 12:55:30.495772 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:30.495606    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"
Oct 30 12:55:30.495772 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:30.495615    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="493dedbc3c8d1fb35fb36420a758d13a375ea3f3ff78862f3a0fd02522a8a7eb"
Oct 30 12:55:31.367704 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.367667    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"]
Oct 30 12:55:31.368498 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.368478    5837 cpu_manager.go:414] "containerMap: removing container" podUID="747e76a9-41d7-4942-a33f-53aa5fd53507" containerName="revoke-keys"
Oct 30 12:55:31.368551 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.368501    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747e76a9-41d7-4942-a33f-53aa5fd53507" containerName="revoke-keys"
Oct 30 12:55:31.368656 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.368646    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="747e76a9-41d7-4942-a33f-53aa5fd53507" containerName="revoke-keys"
Oct 30 12:55:31.372202 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.372186    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"
Oct 30 12:55:31.391452 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.391423    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"]
Oct 30 12:55:31.398098 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.398070    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:55:31.398236 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.398157    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:55:31.398300 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.398264    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:55:31.398588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.398549    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:55:31.408490 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.408463    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1e90ab6-6314-4005-a700-e3012b266236-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7\" (UID: \"c1e90ab6-6314-4005-a700-e3012b266236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"
Oct 30 12:55:31.408735 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.408712    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1e90ab6-6314-4005-a700-e3012b266236-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7\" (UID: \"c1e90ab6-6314-4005-a700-e3012b266236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"
Oct 30 12:55:31.408864 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.408759    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5bb2\" (UniqueName: \"kubernetes.io/projected/c1e90ab6-6314-4005-a700-e3012b266236-kube-api-access-j5bb2\") pod \"maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7\" (UID: \"c1e90ab6-6314-4005-a700-e3012b266236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"
Oct 30 12:55:31.510178 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.510148    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1e90ab6-6314-4005-a700-e3012b266236-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7\" (UID: \"c1e90ab6-6314-4005-a700-e3012b266236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"
Oct 30 12:55:31.510467 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.510444    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1e90ab6-6314-4005-a700-e3012b266236-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7\" (UID: \"c1e90ab6-6314-4005-a700-e3012b266236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"
Oct 30 12:55:31.510525 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.510508    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5bb2\" (UniqueName: \"kubernetes.io/projected/c1e90ab6-6314-4005-a700-e3012b266236-kube-api-access-j5bb2\") pod \"maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7\" (UID: \"c1e90ab6-6314-4005-a700-e3012b266236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"
Oct 30 12:55:31.510887 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.510871    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1e90ab6-6314-4005-a700-e3012b266236-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7\" (UID: \"c1e90ab6-6314-4005-a700-e3012b266236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"
Oct 30 12:55:31.512824 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.512803    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1e90ab6-6314-4005-a700-e3012b266236-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7\" (UID: \"c1e90ab6-6314-4005-a700-e3012b266236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"
Oct 30 12:55:31.530068 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.530050    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5bb2\" (UniqueName: \"kubernetes.io/projected/c1e90ab6-6314-4005-a700-e3012b266236-kube-api-access-j5bb2\") pod \"maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7\" (UID: \"c1e90ab6-6314-4005-a700-e3012b266236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"
Oct 30 12:55:31.686343 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.686276    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"
Oct 30 12:55:31.985855 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:55:31.985817    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e90ab6_6314_4005_a700_e3012b266236.slice/crio-bf0b3e4ab62814ed28b99219f34bf753774623c336378d85d5dc777d9a386dec WatchSource:0}: Error finding container bf0b3e4ab62814ed28b99219f34bf753774623c336378d85d5dc777d9a386dec: Status 404 returned error can't find the container with id bf0b3e4ab62814ed28b99219f34bf753774623c336378d85d5dc777d9a386dec
Oct 30 12:55:31.988517 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:31.988488    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"]
Oct 30 12:55:32.512120 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:32.512084    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7" event={"ID":"c1e90ab6-6314-4005-a700-e3012b266236","Type":"ContainerStarted","Data":"88e4ce5581d3f6d0c663ecf2c054b1def54ca8210d292cef4a44ce9278842cef"}
Oct 30 12:55:32.512530 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:32.512137    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7" event={"ID":"c1e90ab6-6314-4005-a700-e3012b266236","Type":"ContainerStarted","Data":"bf0b3e4ab62814ed28b99219f34bf753774623c336378d85d5dc777d9a386dec"}
Oct 30 12:55:32.551468 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:32.551417    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7" podStartSLOduration=1.5513869150000001 podStartE2EDuration="1.551386915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:32.548836382 +0000 UTC m=+1121.073120601" watchObservedRunningTime="2026-10-30 12:55:32.551386915 +0000 UTC m=+1121.075671133"
Oct 30 12:55:33.521536 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:33.521506    5837 generic.go:356] "Generic (PLEG): container finished" podID="c1e90ab6-6314-4005-a700-e3012b266236" containerID="88e4ce5581d3f6d0c663ecf2c054b1def54ca8210d292cef4a44ce9278842cef" exitCode=0
Oct 30 12:55:33.521936 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:33.521592    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7" event={"ID":"c1e90ab6-6314-4005-a700-e3012b266236","Type":"ContainerDied","Data":"88e4ce5581d3f6d0c663ecf2c054b1def54ca8210d292cef4a44ce9278842cef"}
Oct 30 12:55:34.762342 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:34.761915    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"
Oct 30 12:55:34.901156 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:34.901075    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e90ab6-6314-4005-a700-e3012b266236-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1e90ab6-6314-4005-a700-e3012b266236-maas-api-openshift-ca\") pod \"c1e90ab6-6314-4005-a700-e3012b266236\" (UID: \"c1e90ab6-6314-4005-a700-e3012b266236\") "
Oct 30 12:55:34.901311 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:34.901211    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e90ab6-6314-4005-a700-e3012b266236-kube-api-access-j5bb2\" (UniqueName: \"kubernetes.io/projected/c1e90ab6-6314-4005-a700-e3012b266236-kube-api-access-j5bb2\") pod \"c1e90ab6-6314-4005-a700-e3012b266236\" (UID: \"c1e90ab6-6314-4005-a700-e3012b266236\") "
Oct 30 12:55:34.901311 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:34.901244    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1e90ab6-6314-4005-a700-e3012b266236-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1e90ab6-6314-4005-a700-e3012b266236-maas-api-xks-ca\") pod \"c1e90ab6-6314-4005-a700-e3012b266236\" (UID: \"c1e90ab6-6314-4005-a700-e3012b266236\") "
Oct 30 12:55:34.901451 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:34.901426    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e90ab6-6314-4005-a700-e3012b266236-maas-api-openshift-ca" pod "c1e90ab6-6314-4005-a700-e3012b266236" (UID: "c1e90ab6-6314-4005-a700-e3012b266236"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:34.901876 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:34.901854    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1e90ab6-6314-4005-a700-e3012b266236-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:34.903566 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:34.903543    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1e90ab6-6314-4005-a700-e3012b266236-maas-api-xks-ca" pod "c1e90ab6-6314-4005-a700-e3012b266236" (UID: "c1e90ab6-6314-4005-a700-e3012b266236"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:55:34.903629 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:34.903582    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e90ab6-6314-4005-a700-e3012b266236-kube-api-access-j5bb2" pod "c1e90ab6-6314-4005-a700-e3012b266236" (UID: "c1e90ab6-6314-4005-a700-e3012b266236"). InnerVolumeSpecName "kube-api-access-j5bb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:35.003256 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.003228    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5bb2\" (UniqueName: \"kubernetes.io/projected/c1e90ab6-6314-4005-a700-e3012b266236-kube-api-access-j5bb2\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:35.003256 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.003254    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1e90ab6-6314-4005-a700-e3012b266236-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:35.461996 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.461974    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:55:35.542782 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.542752    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"
Oct 30 12:55:35.542975 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.542783    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7" event={"ID":"c1e90ab6-6314-4005-a700-e3012b266236","Type":"ContainerDied","Data":"bf0b3e4ab62814ed28b99219f34bf753774623c336378d85d5dc777d9a386dec"}
Oct 30 12:55:35.542975 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.542839    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf0b3e4ab62814ed28b99219f34bf753774623c336378d85d5dc777d9a386dec"
Oct 30 12:55:35.545127 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.545099    5837 generic.go:356] "Generic (PLEG): container finished" podID="1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" containerID="ba556f50c847aa6ffee2143e395b5783af0d19246baae41e91c215feccc67399" exitCode=0
Oct 30 12:55:35.545247 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.545179    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"
Oct 30 12:55:35.545323 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.545175    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s" event={"ID":"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d","Type":"ContainerDied","Data":"ba556f50c847aa6ffee2143e395b5783af0d19246baae41e91c215feccc67399"}
Oct 30 12:55:35.545323 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.545304    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s" event={"ID":"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d","Type":"ContainerDied","Data":"75c866bc3f582ac7e2980de4f629c1de3e29f5a1986a2546ad22adac05da9e4e"}
Oct 30 12:55:35.545462 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.545339    5837 scope.go:122] "RemoveContainer" containerID="ba556f50c847aa6ffee2143e395b5783af0d19246baae41e91c215feccc67399"
Oct 30 12:55:35.558859 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.558842    5837 scope.go:122] "RemoveContainer" containerID="ba556f50c847aa6ffee2143e395b5783af0d19246baae41e91c215feccc67399"
Oct 30 12:55:35.559190 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:55:35.559167    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba556f50c847aa6ffee2143e395b5783af0d19246baae41e91c215feccc67399\": container with ID starting with ba556f50c847aa6ffee2143e395b5783af0d19246baae41e91c215feccc67399 not found: ID does not exist" containerID="ba556f50c847aa6ffee2143e395b5783af0d19246baae41e91c215feccc67399"
Oct 30 12:55:35.559248 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.559202    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba556f50c847aa6ffee2143e395b5783af0d19246baae41e91c215feccc67399"} err="failed to get container status \"ba556f50c847aa6ffee2143e395b5783af0d19246baae41e91c215feccc67399\": rpc error: code = NotFound desc = could not find container \"ba556f50c847aa6ffee2143e395b5783af0d19246baae41e91c215feccc67399\": container with ID starting with ba556f50c847aa6ffee2143e395b5783af0d19246baae41e91c215feccc67399 not found: ID does not exist"
Oct 30 12:55:35.611151 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.611126    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istiod-ca-cert\") pod \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") "
Oct 30 12:55:35.611301 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.611229    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-data\") pod \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") "
Oct 30 12:55:35.611375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.611323    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-kube-api-access-x9d9d\" (UniqueName: \"kubernetes.io/projected/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-kube-api-access-x9d9d\") pod \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") "
Oct 30 12:55:35.611484 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.611465    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-token\" (UniqueName: \"kubernetes.io/projected/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-token\") pod \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") "
Oct 30 12:55:35.611551 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.611521    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-workload-socket\") pod \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") "
Oct 30 12:55:35.611551 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.611526    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-data" pod "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" (UID: "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:35.611657 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.611572    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istiod-ca-cert" pod "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" (UID: "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:35.611657 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.611598    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-envoy\") pod \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") "
Oct 30 12:55:35.611657 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.611647    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-podinfo\") pod \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") "
Oct 30 12:55:35.611841 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.611720    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-credential-socket\") pod \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") "
Oct 30 12:55:35.611841 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.611791    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-workload-certs\") pod \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\" (UID: \"1d98825f-f663-4f10-be1a-ce6e6ecc0c3d\") "
Oct 30 12:55:35.611942 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.611832    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-workload-socket" pod "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" (UID: "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:35.612087 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.612064    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-workload-certs" pod "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" (UID: "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:35.612087 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.612077    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-credential-socket" pod "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" (UID: "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:35.612338 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.612301    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:35.612338 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.612324    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:35.612473 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.612339    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:35.612473 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.612353    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:35.612473 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.612365    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:35.614092 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.614064    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-podinfo" pod "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" (UID: "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:55:35.614219 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.614185    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-kube-api-access-x9d9d" pod "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" (UID: "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d"). InnerVolumeSpecName "kube-api-access-x9d9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:35.614411 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.614371    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-token" pod "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" (UID: "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:35.614492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.614449    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-envoy" pod "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" (UID: "1d98825f-f663-4f10-be1a-ce6e6ecc0c3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:35.713532 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.713491    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9d9d\" (UniqueName: \"kubernetes.io/projected/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-kube-api-access-x9d9d\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:35.713532 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.713531    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:35.713748 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.713547    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:35.713748 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.713560    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:55:35.891502 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.891479    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"]
Oct 30 12:55:35.895700 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:35.895674    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-rmq6s"]
Oct 30 12:55:37.606246 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:37.606217    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" path="/var/lib/kubelet/pods/1d98825f-f663-4f10-be1a-ce6e6ecc0c3d/volumes"
Oct 30 12:55:45.095381 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:55:45.095342    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:55:55.716921 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:55.716891    5837 scope.go:122] "RemoveContainer" containerID="4af1f75df7231c0e554793fcef59086760c9edabc1be3d5f91348b2ec7389698"
Oct 30 12:55:55.808101 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:55.808063    5837 scope.go:122] "RemoveContainer" containerID="39c6271bd09fe83604889ab39058891cd3b1849dfb606f4c782199bbb47e4f07"
Oct 30 12:55:55.862945 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:55.862918    5837 scope.go:122] "RemoveContainer" containerID="79a0764d0b05f73db56629cc269696b9f236a18ef3d298057b035576fd7f14bf"
Oct 30 12:55:55.873344 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:55.873324    5837 scope.go:122] "RemoveContainer" containerID="ffefadfca48e5f0f2c351e5cf035ac2d660e865802b3c2ab538c81a03e3a1a9b"
Oct 30 12:55:55.882219 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:55.882199    5837 scope.go:122] "RemoveContainer" containerID="615e389c883408a88bc9a8bc6dbbe7b1c637facd5e80d5f81b445df6d0cde591"
Oct 30 12:55:55.891444 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:55:55.891426    5837 scope.go:122] "RemoveContainer" containerID="353d0110cebecfebe75120b0fe58eca60de4750974ae82d3117307016f9b2058"
Oct 30 12:56:06.525561 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:06.525512    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:56:06.540422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:06.535491    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:07.095420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:07.094495    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:07.529221 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:07.529160    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:56:07.586470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:07.575512    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:56:07.586470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:07.556039    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:56:07.587829 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:07.586842    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:56:08.124421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:08.110905    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:08.152105 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:08.152076    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:56:08.631426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:08.626864    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:08.737463 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:08.722507    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:09.162421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:09.157531    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:56:09.746423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:09.726625    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": context deadline exceeded"
Oct 30 12:56:09.754416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:09.750505    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:09.996421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:09.995648    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:56:10.275818 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:10.275764    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:56:10.543557 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:10.543431    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"]
Oct 30 12:56:10.588095 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:10.588070    5837 cpu_manager.go:414] "containerMap: removing container" podUID="c1e90ab6-6314-4005-a700-e3012b266236" containerName="revoke-keys"
Oct 30 12:56:10.588337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:10.588322    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e90ab6-6314-4005-a700-e3012b266236" containerName="revoke-keys"
Oct 30 12:56:10.588587 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:10.588575    5837 cpu_manager.go:414] "containerMap: removing container" podUID="1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" containerName="istio-proxy"
Oct 30 12:56:10.588701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:10.588689    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" containerName="istio-proxy"
Oct 30 12:56:10.589044 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:10.589029    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d98825f-f663-4f10-be1a-ce6e6ecc0c3d" containerName="istio-proxy"
Oct 30 12:56:10.592470 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:10.592449    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e90ab6-6314-4005-a700-e3012b266236" containerName="revoke-keys"
Oct 30 12:56:10.616498 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:10.613784    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:10.872476 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:10.864623    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"]
Oct 30 12:56:10.872476 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:10.865110    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:10.872476 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:10.867539    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:56:10.922442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:10.894739    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:56:10.922442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:10.894798    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" podUID="faf87439-5044-43cd-9b1a-200188299f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:10.922442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:10.916766    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ef1186-openshift-default-dockercfg-vccv5\""
Oct 30 12:56:11.060422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.046696    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:11.069742 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.069705    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:56:11.070030 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.069998    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:11.122738 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.122549    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:56:11.122738 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.122619    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:11.139424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.133550    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:56:11.139424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.133648    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" podUID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:11.199421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.173855    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467baab9-bc62-4c3f-a034-651a6491271c-istio-podinfo\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.199421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.173950    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-istio-envoy\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.199421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.173995    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-istio-data\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.199421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.174075    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467baab9-bc62-4c3f-a034-651a6491271c-istiod-ca-cert\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.199421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.174240    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-credential-socket\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.199421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.174275    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-workload-certs\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.199421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.174349    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-workload-socket\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.199421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.174376    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqwwg\" (UniqueName: \"kubernetes.io/projected/467baab9-bc62-4c3f-a034-651a6491271c-kube-api-access-dqwwg\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.199421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.174440    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467baab9-bc62-4c3f-a034-651a6491271c-istio-token\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.276463 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.276310    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-istio-envoy\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.276463 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.276382    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-istio-data\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.276463 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.276457    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467baab9-bc62-4c3f-a034-651a6491271c-istiod-ca-cert\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.276770 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.276577    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-credential-socket\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.276770 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.276604    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-workload-certs\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.276770 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.276659    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-workload-socket\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.276770 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.276686    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwwg\" (UniqueName: \"kubernetes.io/projected/467baab9-bc62-4c3f-a034-651a6491271c-kube-api-access-dqwwg\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.276770 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.276724    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467baab9-bc62-4c3f-a034-651a6491271c-istio-token\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.277084 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.276882    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467baab9-bc62-4c3f-a034-651a6491271c-istio-podinfo\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.278440 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.278382    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-credential-socket\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.325417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.308802    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467baab9-bc62-4c3f-a034-651a6491271c-istiod-ca-cert\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.325417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.319048    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-workload-socket\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.325417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.319200    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:11.325417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.321945    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-istio-envoy\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.327334 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.327279    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-workload-certs\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.327748 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.327723    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-istio-data\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.340112 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.340078    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqwwg\" (UniqueName: \"kubernetes.io/projected/467baab9-bc62-4c3f-a034-651a6491271c-kube-api-access-dqwwg\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.349077 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.349010    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467baab9-bc62-4c3f-a034-651a6491271c-istio-podinfo\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.520423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.520072    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467baab9-bc62-4c3f-a034-651a6491271c-istio-token\") pod \"e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") " pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.755463 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.755429    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:11.886011 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:11.885968    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:12.054481 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:12.047847    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:12.324973 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:12.324879    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:12.907697 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:12.906593    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:13.050674 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:13.050628    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:13.349714 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:13.349630    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:56:13.574461 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:56:13.571270    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467baab9_bc62_4c3f_a034_651a6491271c.slice/crio-c9fbf0daa8087246cd3fac1298f97e682bd51d324475d9296fe041d7e01befb7 WatchSource:0}: Error finding container c9fbf0daa8087246cd3fac1298f97e682bd51d324475d9296fe041d7e01befb7: Status 404 returned error can't find the container with id c9fbf0daa8087246cd3fac1298f97e682bd51d324475d9296fe041d7e01befb7
Oct 30 12:56:13.627420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:13.623845    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:13.652216 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:13.652167    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:13.652554 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:13.652530    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:13.933140 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:13.933042    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:14.029461 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:14.023137    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"]
Oct 30 12:56:14.235489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:14.092153    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:14.379422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:14.375471    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" event={"ID":"467baab9-bc62-4c3f-a034-651a6491271c","Type":"ContainerStarted","Data":"c9fbf0daa8087246cd3fac1298f97e682bd51d324475d9296fe041d7e01befb7"}
Oct 30 12:56:14.578689 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:14.379730    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:56:14.578960 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:14.485910    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": context deadline exceeded"
Oct 30 12:56:14.898516 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:14.898378    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:56:14.974605 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:14.974565    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:56:15.008318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:14.982209    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:56:15.151170 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:15.150260    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:56:15.276521 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:15.276482    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" event={"ID":"467baab9-bc62-4c3f-a034-651a6491271c","Type":"ContainerStarted","Data":"6a55cad6a1616762cea51709c2c8eb27400cd6310ee82b191295227aa036ccea"}
Oct 30 12:56:15.371230 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:56:15.371142    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:56:15.937016 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:15.936972    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"]
Oct 30 12:56:15.949372 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:15.949332    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a531c-nd2w7"]
Oct 30 12:56:16.372304 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:16.372249    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" podStartSLOduration=6.372235224 podStartE2EDuration="6.372235224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:16.369598856 +0000 UTC m=+1164.893883075" watchObservedRunningTime="2026-10-30 12:56:16.372235224 +0000 UTC m=+1164.896519442"
Oct 30 12:56:16.759647 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:16.759619    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:16.777069 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:16.777036    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:17.335557 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:17.335526    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:17.336588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:17.336564    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:56:17.629734 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:17.629631    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e90ab6-6314-4005-a700-e3012b266236" path="/var/lib/kubelet/pods/c1e90ab6-6314-4005-a700-e3012b266236/volumes"
Oct 30 12:56:22.383368 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:22.383288    5837 generic.go:356] "Generic (PLEG): container finished" podID="d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" containerID="b92a811e43f026fa57e988fd33f822abda88ba9b84353a6cc22cdbf3738933f9" exitCode=0
Oct 30 12:56:22.383368 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:22.383356    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5" event={"ID":"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113","Type":"ContainerDied","Data":"b92a811e43f026fa57e988fd33f822abda88ba9b84353a6cc22cdbf3738933f9"}
Oct 30 12:56:22.383892 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:22.383878    5837 scope.go:122] "RemoveContainer" containerID="b92a811e43f026fa57e988fd33f822abda88ba9b84353a6cc22cdbf3738933f9"
Oct 30 12:56:22.385570 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:22.385540    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:22.385691 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:22.385625    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:22.385691 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:22.385675    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:22.709226 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:22.709136    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"]
Oct 30 12:56:22.709667 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:22.709636    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy" containerID="cri-o://7db390e27e9a0dce6541a6f06d607836e7c7f35686073bb3475bfdbc60106ad4" gracePeriod=30
Oct 30 12:56:23.319177 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.319144    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"]
Oct 30 12:56:23.323210 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.323183    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"
Oct 30 12:56:23.341854 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.341827    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"]
Oct 30 12:56:23.347789 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.347764    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:56:23.347936 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.347822    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:56:23.347936 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.347875    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:56:23.349469 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.349447    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:56:23.399689 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.399650    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5" event={"ID":"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113","Type":"ContainerStarted","Data":"4e4f77c9e3572a67ac3665d35bc445bfb6f4e71d412e66fe591ff12cc0ea0684"}
Oct 30 12:56:23.442932 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.442901    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f5c0a1c-62f7-4e90-967a-1c8461363273-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb\" (UID: \"0f5c0a1c-62f7-4e90-967a-1c8461363273\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"
Oct 30 12:56:23.443145 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.443013    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f5c0a1c-62f7-4e90-967a-1c8461363273-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb\" (UID: \"0f5c0a1c-62f7-4e90-967a-1c8461363273\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"
Oct 30 12:56:23.443278 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.443234    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjs9q\" (UniqueName: \"kubernetes.io/projected/0f5c0a1c-62f7-4e90-967a-1c8461363273-kube-api-access-cjs9q\") pod \"maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb\" (UID: \"0f5c0a1c-62f7-4e90-967a-1c8461363273\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"
Oct 30 12:56:23.544739 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.544708    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f5c0a1c-62f7-4e90-967a-1c8461363273-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb\" (UID: \"0f5c0a1c-62f7-4e90-967a-1c8461363273\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"
Oct 30 12:56:23.544940 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.544836    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f5c0a1c-62f7-4e90-967a-1c8461363273-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb\" (UID: \"0f5c0a1c-62f7-4e90-967a-1c8461363273\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"
Oct 30 12:56:23.544940 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.544903    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjs9q\" (UniqueName: \"kubernetes.io/projected/0f5c0a1c-62f7-4e90-967a-1c8461363273-kube-api-access-cjs9q\") pod \"maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb\" (UID: \"0f5c0a1c-62f7-4e90-967a-1c8461363273\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"
Oct 30 12:56:23.546097 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.546071    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f5c0a1c-62f7-4e90-967a-1c8461363273-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb\" (UID: \"0f5c0a1c-62f7-4e90-967a-1c8461363273\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"
Oct 30 12:56:23.547532 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.547499    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f5c0a1c-62f7-4e90-967a-1c8461363273-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb\" (UID: \"0f5c0a1c-62f7-4e90-967a-1c8461363273\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"
Oct 30 12:56:23.564742 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.564711    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjs9q\" (UniqueName: \"kubernetes.io/projected/0f5c0a1c-62f7-4e90-967a-1c8461363273-kube-api-access-cjs9q\") pod \"maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb\" (UID: \"0f5c0a1c-62f7-4e90-967a-1c8461363273\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"
Oct 30 12:56:23.639246 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.639162    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"
Oct 30 12:56:23.973741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:23.973706    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"]
Oct 30 12:56:23.974281 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:56:23.974259    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5c0a1c_62f7_4e90_967a_1c8461363273.slice/crio-4dc1f01d1106758a53101c40fce658d07d446e5aa5e69f8223ee36c720ddcf37 WatchSource:0}: Error finding container 4dc1f01d1106758a53101c40fce658d07d446e5aa5e69f8223ee36c720ddcf37: Status 404 returned error can't find the container with id 4dc1f01d1106758a53101c40fce658d07d446e5aa5e69f8223ee36c720ddcf37
Oct 30 12:56:24.293296 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:24.293254    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:56:24.293736 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:24.293708    5837 patch_prober.go:29] interesting pod/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:56:24.293957 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:24.293914    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5" podUID="d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 12:56:24.417371 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:24.417337    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb" event={"ID":"0f5c0a1c-62f7-4e90-967a-1c8461363273","Type":"ContainerStarted","Data":"4dc1f01d1106758a53101c40fce658d07d446e5aa5e69f8223ee36c720ddcf37"}
Oct 30 12:56:25.305420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:25.302896    5837 patch_prober.go:29] interesting pod/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:56:25.305420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:25.302960    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5" podUID="d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 12:56:25.425505 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:25.425382    5837 generic.go:356] "Generic (PLEG): container finished" podID="0f5c0a1c-62f7-4e90-967a-1c8461363273" containerID="ab809a0f3fee82a625cc2e492d17b2d2859d838fbe21665f3c1192b171d34240" exitCode=0
Oct 30 12:56:25.425505 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:25.425473    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb" event={"ID":"0f5c0a1c-62f7-4e90-967a-1c8461363273","Type":"ContainerDied","Data":"ab809a0f3fee82a625cc2e492d17b2d2859d838fbe21665f3c1192b171d34240"}
Oct 30 12:56:26.297430 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:26.292669    5837 patch_prober.go:29] interesting pod/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:56:26.297430 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:26.292729    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5" podUID="d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 12:56:26.622639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:26.622607    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"
Oct 30 12:56:26.633058 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:26.632975    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f5c0a1c-62f7-4e90-967a-1c8461363273-kube-api-access-cjs9q\" (UniqueName: \"kubernetes.io/projected/0f5c0a1c-62f7-4e90-967a-1c8461363273-kube-api-access-cjs9q\") pod \"0f5c0a1c-62f7-4e90-967a-1c8461363273\" (UID: \"0f5c0a1c-62f7-4e90-967a-1c8461363273\") "
Oct 30 12:56:26.633058 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:26.633041    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f5c0a1c-62f7-4e90-967a-1c8461363273-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f5c0a1c-62f7-4e90-967a-1c8461363273-maas-api-xks-ca\") pod \"0f5c0a1c-62f7-4e90-967a-1c8461363273\" (UID: \"0f5c0a1c-62f7-4e90-967a-1c8461363273\") "
Oct 30 12:56:26.633288 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:26.633265    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f5c0a1c-62f7-4e90-967a-1c8461363273-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f5c0a1c-62f7-4e90-967a-1c8461363273-maas-api-openshift-ca\") pod \"0f5c0a1c-62f7-4e90-967a-1c8461363273\" (UID: \"0f5c0a1c-62f7-4e90-967a-1c8461363273\") "
Oct 30 12:56:26.633647 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:26.633624    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f5c0a1c-62f7-4e90-967a-1c8461363273-maas-api-openshift-ca" pod "0f5c0a1c-62f7-4e90-967a-1c8461363273" (UID: "0f5c0a1c-62f7-4e90-967a-1c8461363273"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:56:26.633892 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:26.633869    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f5c0a1c-62f7-4e90-967a-1c8461363273-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:26.636008 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:26.635987    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5c0a1c-62f7-4e90-967a-1c8461363273-kube-api-access-cjs9q" pod "0f5c0a1c-62f7-4e90-967a-1c8461363273" (UID: "0f5c0a1c-62f7-4e90-967a-1c8461363273"). InnerVolumeSpecName "kube-api-access-cjs9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:56:26.636415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:26.636381    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f5c0a1c-62f7-4e90-967a-1c8461363273-maas-api-xks-ca" pod "0f5c0a1c-62f7-4e90-967a-1c8461363273" (UID: "0f5c0a1c-62f7-4e90-967a-1c8461363273"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:56:26.735830 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:26.735784    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjs9q\" (UniqueName: \"kubernetes.io/projected/0f5c0a1c-62f7-4e90-967a-1c8461363273-kube-api-access-cjs9q\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:26.735830 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:26.735829    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f5c0a1c-62f7-4e90-967a-1c8461363273-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:27.292276 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:27.291801    5837 patch_prober.go:29] interesting pod/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:56:27.292276 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:27.291864    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5" podUID="d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 12:56:27.441770 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:27.441740    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"
Oct 30 12:56:27.441770 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:27.441756    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb" event={"ID":"0f5c0a1c-62f7-4e90-967a-1c8461363273","Type":"ContainerDied","Data":"4dc1f01d1106758a53101c40fce658d07d446e5aa5e69f8223ee36c720ddcf37"}
Oct 30 12:56:27.441953 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:27.441811    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dc1f01d1106758a53101c40fce658d07d446e5aa5e69f8223ee36c720ddcf37"
Oct 30 12:56:28.309420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.308805    5837 patch_prober.go:29] interesting pod/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 12:56:28.309420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.308869    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 12:56:28.311420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.310224    5837 patch_prober.go:29] interesting pod/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:56:28.311420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.310280    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5" podUID="d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 12:56:28.445744 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.445721    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:56:28.450176 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.450146    5837 generic.go:356] "Generic (PLEG): container finished" podID="484f0dd9-59d8-4690-a051-044c1fed4945" containerID="7db390e27e9a0dce6541a6f06d607836e7c7f35686073bb3475bfdbc60106ad4" exitCode=0
Oct 30 12:56:28.450298 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.450200    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"
Oct 30 12:56:28.450298 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.450244    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" event={"ID":"484f0dd9-59d8-4690-a051-044c1fed4945","Type":"ContainerDied","Data":"7db390e27e9a0dce6541a6f06d607836e7c7f35686073bb3475bfdbc60106ad4"}
Oct 30 12:56:28.450420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.450344    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk" event={"ID":"484f0dd9-59d8-4690-a051-044c1fed4945","Type":"ContainerDied","Data":"032ce4df6a308efca01feb283be3780abb6babdd4c09768d42930e5b0fa733c0"}
Oct 30 12:56:28.450420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.450381    5837 scope.go:122] "RemoveContainer" containerID="7db390e27e9a0dce6541a6f06d607836e7c7f35686073bb3475bfdbc60106ad4"
Oct 30 12:56:28.465161 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.465133    5837 scope.go:122] "RemoveContainer" containerID="85593eac598b59dc3265222df1e10753f93785187b65e244934970880951cd99"
Oct 30 12:56:28.476361 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.476342    5837 scope.go:122] "RemoveContainer" containerID="7db390e27e9a0dce6541a6f06d607836e7c7f35686073bb3475bfdbc60106ad4"
Oct 30 12:56:28.476683 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:56:28.476663    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db390e27e9a0dce6541a6f06d607836e7c7f35686073bb3475bfdbc60106ad4\": container with ID starting with 7db390e27e9a0dce6541a6f06d607836e7c7f35686073bb3475bfdbc60106ad4 not found: ID does not exist" containerID="7db390e27e9a0dce6541a6f06d607836e7c7f35686073bb3475bfdbc60106ad4"
Oct 30 12:56:28.476749 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.476692    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db390e27e9a0dce6541a6f06d607836e7c7f35686073bb3475bfdbc60106ad4"} err="failed to get container status \"7db390e27e9a0dce6541a6f06d607836e7c7f35686073bb3475bfdbc60106ad4\": rpc error: code = NotFound desc = could not find container \"7db390e27e9a0dce6541a6f06d607836e7c7f35686073bb3475bfdbc60106ad4\": container with ID starting with 7db390e27e9a0dce6541a6f06d607836e7c7f35686073bb3475bfdbc60106ad4 not found: ID does not exist"
Oct 30 12:56:28.476749 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.476711    5837 scope.go:122] "RemoveContainer" containerID="85593eac598b59dc3265222df1e10753f93785187b65e244934970880951cd99"
Oct 30 12:56:28.476980 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:56:28.476948    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85593eac598b59dc3265222df1e10753f93785187b65e244934970880951cd99\": container with ID starting with 85593eac598b59dc3265222df1e10753f93785187b65e244934970880951cd99 not found: ID does not exist" containerID="85593eac598b59dc3265222df1e10753f93785187b65e244934970880951cd99"
Oct 30 12:56:28.477030 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.476985    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85593eac598b59dc3265222df1e10753f93785187b65e244934970880951cd99"} err="failed to get container status \"85593eac598b59dc3265222df1e10753f93785187b65e244934970880951cd99\": rpc error: code = NotFound desc = could not find container \"85593eac598b59dc3265222df1e10753f93785187b65e244934970880951cd99\": container with ID starting with 85593eac598b59dc3265222df1e10753f93785187b65e244934970880951cd99 not found: ID does not exist"
Oct 30 12:56:28.480501 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.480483    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-istio-data\") pod \"484f0dd9-59d8-4690-a051-044c1fed4945\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") "
Oct 30 12:56:28.480696 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.480683    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484f0dd9-59d8-4690-a051-044c1fed4945-istio-token\" (UniqueName: \"kubernetes.io/projected/484f0dd9-59d8-4690-a051-044c1fed4945-istio-token\") pod \"484f0dd9-59d8-4690-a051-044c1fed4945\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") "
Oct 30 12:56:28.480772 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.480716    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-istio-envoy\") pod \"484f0dd9-59d8-4690-a051-044c1fed4945\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") "
Oct 30 12:56:28.480772 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.480735    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/484f0dd9-59d8-4690-a051-044c1fed4945-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484f0dd9-59d8-4690-a051-044c1fed4945-istio-podinfo\") pod \"484f0dd9-59d8-4690-a051-044c1fed4945\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") "
Oct 30 12:56:28.480772 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.480731    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-istio-data" pod "484f0dd9-59d8-4690-a051-044c1fed4945" (UID: "484f0dd9-59d8-4690-a051-044c1fed4945"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:56:28.480934 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.480855    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/484f0dd9-59d8-4690-a051-044c1fed4945-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484f0dd9-59d8-4690-a051-044c1fed4945-istiod-ca-cert\") pod \"484f0dd9-59d8-4690-a051-044c1fed4945\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") "
Oct 30 12:56:28.480934 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.480910    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-workload-certs\") pod \"484f0dd9-59d8-4690-a051-044c1fed4945\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") "
Oct 30 12:56:28.481028 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.480949    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-workload-socket\") pod \"484f0dd9-59d8-4690-a051-044c1fed4945\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") "
Oct 30 12:56:28.481028 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.481005    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-credential-socket\") pod \"484f0dd9-59d8-4690-a051-044c1fed4945\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") "
Oct 30 12:56:28.481121 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.481077    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484f0dd9-59d8-4690-a051-044c1fed4945-kube-api-access-vzgsm\" (UniqueName: \"kubernetes.io/projected/484f0dd9-59d8-4690-a051-044c1fed4945-kube-api-access-vzgsm\") pod \"484f0dd9-59d8-4690-a051-044c1fed4945\" (UID: \"484f0dd9-59d8-4690-a051-044c1fed4945\") "
Oct 30 12:56:28.481232 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.481205    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484f0dd9-59d8-4690-a051-044c1fed4945-istiod-ca-cert" pod "484f0dd9-59d8-4690-a051-044c1fed4945" (UID: "484f0dd9-59d8-4690-a051-044c1fed4945"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:56:28.481306 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.481278    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-credential-socket" pod "484f0dd9-59d8-4690-a051-044c1fed4945" (UID: "484f0dd9-59d8-4690-a051-044c1fed4945"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:56:28.481361 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.481233    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-workload-certs" pod "484f0dd9-59d8-4690-a051-044c1fed4945" (UID: "484f0dd9-59d8-4690-a051-044c1fed4945"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:56:28.481600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.481576    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-workload-socket" pod "484f0dd9-59d8-4690-a051-044c1fed4945" (UID: "484f0dd9-59d8-4690-a051-044c1fed4945"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:56:28.481843 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.481829    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:28.481943 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.481933    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:28.482001 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.481994    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484f0dd9-59d8-4690-a051-044c1fed4945-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:28.482065 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.482058    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:28.482118 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.482111    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:28.499422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.483467    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484f0dd9-59d8-4690-a051-044c1fed4945-kube-api-access-vzgsm" pod "484f0dd9-59d8-4690-a051-044c1fed4945" (UID: "484f0dd9-59d8-4690-a051-044c1fed4945"). InnerVolumeSpecName "kube-api-access-vzgsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:56:28.509360 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.508542    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-istio-envoy" pod "484f0dd9-59d8-4690-a051-044c1fed4945" (UID: "484f0dd9-59d8-4690-a051-044c1fed4945"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:56:28.518367 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.515547    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484f0dd9-59d8-4690-a051-044c1fed4945-istio-token" pod "484f0dd9-59d8-4690-a051-044c1fed4945" (UID: "484f0dd9-59d8-4690-a051-044c1fed4945"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:56:28.518367 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.518327    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/484f0dd9-59d8-4690-a051-044c1fed4945-istio-podinfo" pod "484f0dd9-59d8-4690-a051-044c1fed4945" (UID: "484f0dd9-59d8-4690-a051-044c1fed4945"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:56:28.583408 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.583368    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzgsm\" (UniqueName: \"kubernetes.io/projected/484f0dd9-59d8-4690-a051-044c1fed4945-kube-api-access-vzgsm\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:28.583408 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.583415    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484f0dd9-59d8-4690-a051-044c1fed4945-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:28.583671 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.583433    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484f0dd9-59d8-4690-a051-044c1fed4945-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:28.583671 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.583445    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484f0dd9-59d8-4690-a051-044c1fed4945-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:28.820241 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.820152    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"]
Oct 30 12:56:28.828515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:28.828483    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0a531c-openshift-default-585b78df5-lwjzk"]
Oct 30 12:56:29.425582 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:29.425551    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:56:29.426078 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:29.426059    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:56:29.427516 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:29.427486    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 12:56:29.605653 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:29.605625    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" path="/var/lib/kubelet/pods/484f0dd9-59d8-4690-a051-044c1fed4945/volumes"
Oct 30 12:56:33.649973 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.649903    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"]
Oct 30 12:56:33.650568 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.650554    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0f5c0a1c-62f7-4e90-967a-1c8461363273" containerName="revoke-keys"
Oct 30 12:56:33.650639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.650570    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f5c0a1c-62f7-4e90-967a-1c8461363273" containerName="revoke-keys"
Oct 30 12:56:33.650639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.650590    5837 cpu_manager.go:414] "containerMap: removing container" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy"
Oct 30 12:56:33.650639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.650596    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy"
Oct 30 12:56:33.650639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.650606    5837 cpu_manager.go:414] "containerMap: removing container" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy"
Oct 30 12:56:33.650639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.650611    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy"
Oct 30 12:56:33.650796 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.650727    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy"
Oct 30 12:56:33.650796 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.650744    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f5c0a1c-62f7-4e90-967a-1c8461363273" containerName="revoke-keys"
Oct 30 12:56:33.654418 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.654378    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"
Oct 30 12:56:33.672768 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.672732    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"]
Oct 30 12:56:33.679584 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.679559    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:56:33.679767 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.679595    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:56:33.679767 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.679625    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:56:33.681062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.681041    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:56:33.787990 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.787963    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b2tf\" (UniqueName: \"kubernetes.io/projected/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-kube-api-access-4b2tf\") pod \"maas-api-revoke-keys-e2e-api-a-6bda55-f49zq\" (UID: \"e227fd1a-7bd2-4ad0-a593-09f1f5c06537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"
Oct 30 12:56:33.788156 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.788105    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6bda55-f49zq\" (UID: \"e227fd1a-7bd2-4ad0-a593-09f1f5c06537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"
Oct 30 12:56:33.788271 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.788252    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6bda55-f49zq\" (UID: \"e227fd1a-7bd2-4ad0-a593-09f1f5c06537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"
Oct 30 12:56:33.889941 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.889912    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6bda55-f49zq\" (UID: \"e227fd1a-7bd2-4ad0-a593-09f1f5c06537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"
Oct 30 12:56:33.890106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.890031    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6bda55-f49zq\" (UID: \"e227fd1a-7bd2-4ad0-a593-09f1f5c06537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"
Oct 30 12:56:33.890172 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.890125    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b2tf\" (UniqueName: \"kubernetes.io/projected/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-kube-api-access-4b2tf\") pod \"maas-api-revoke-keys-e2e-api-a-6bda55-f49zq\" (UID: \"e227fd1a-7bd2-4ad0-a593-09f1f5c06537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"
Oct 30 12:56:33.890699 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.890675    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6bda55-f49zq\" (UID: \"e227fd1a-7bd2-4ad0-a593-09f1f5c06537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"
Oct 30 12:56:33.892662 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.892644    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6bda55-f49zq\" (UID: \"e227fd1a-7bd2-4ad0-a593-09f1f5c06537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"
Oct 30 12:56:33.909165 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.909096    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b2tf\" (UniqueName: \"kubernetes.io/projected/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-kube-api-access-4b2tf\") pod \"maas-api-revoke-keys-e2e-api-a-6bda55-f49zq\" (UID: \"e227fd1a-7bd2-4ad0-a593-09f1f5c06537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"
Oct 30 12:56:33.966960 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:33.966924    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"
Oct 30 12:56:34.169808 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:34.169729    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"]
Oct 30 12:56:34.204266 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:34.204234    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"]
Oct 30 12:56:34.303167 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:34.303127    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"]
Oct 30 12:56:34.498645 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:34.498604    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq" event={"ID":"e227fd1a-7bd2-4ad0-a593-09f1f5c06537","Type":"ContainerStarted","Data":"4685beb614e209614db8d1b36d82c4db8eec9b1004dbaa5f3f76d71985db410c"}
Oct 30 12:56:35.559426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:35.558994    5837 generic.go:356] "Generic (PLEG): container finished" podID="e227fd1a-7bd2-4ad0-a593-09f1f5c06537" containerID="9f688b4495f602fbcc2dfbcaa21c1ae2fd71ba9fa08aafd74de7377a06142d8a" exitCode=0
Oct 30 12:56:35.559426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:35.559171    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq" event={"ID":"e227fd1a-7bd2-4ad0-a593-09f1f5c06537","Type":"ContainerDied","Data":"9f688b4495f602fbcc2dfbcaa21c1ae2fd71ba9fa08aafd74de7377a06142d8a"}
Oct 30 12:56:36.741003 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:36.740969    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"
Oct 30 12:56:36.772664 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:36.772641    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-kube-api-access-4b2tf\" (UniqueName: \"kubernetes.io/projected/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-kube-api-access-4b2tf\") pod \"e227fd1a-7bd2-4ad0-a593-09f1f5c06537\" (UID: \"e227fd1a-7bd2-4ad0-a593-09f1f5c06537\") "
Oct 30 12:56:36.772789 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:36.772766    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-maas-api-openshift-ca\") pod \"e227fd1a-7bd2-4ad0-a593-09f1f5c06537\" (UID: \"e227fd1a-7bd2-4ad0-a593-09f1f5c06537\") "
Oct 30 12:56:36.772831 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:36.772790    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-maas-api-xks-ca\") pod \"e227fd1a-7bd2-4ad0-a593-09f1f5c06537\" (UID: \"e227fd1a-7bd2-4ad0-a593-09f1f5c06537\") "
Oct 30 12:56:36.773072 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:36.773048    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-maas-api-openshift-ca" pod "e227fd1a-7bd2-4ad0-a593-09f1f5c06537" (UID: "e227fd1a-7bd2-4ad0-a593-09f1f5c06537"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:56:36.773497 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:36.773479    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:36.774961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:36.774934    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-maas-api-xks-ca" pod "e227fd1a-7bd2-4ad0-a593-09f1f5c06537" (UID: "e227fd1a-7bd2-4ad0-a593-09f1f5c06537"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:56:36.775048 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:36.774976    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-kube-api-access-4b2tf" pod "e227fd1a-7bd2-4ad0-a593-09f1f5c06537" (UID: "e227fd1a-7bd2-4ad0-a593-09f1f5c06537"). InnerVolumeSpecName "kube-api-access-4b2tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:56:36.875042 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:36.874971    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:36.875042 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:36.874998    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b2tf\" (UniqueName: \"kubernetes.io/projected/e227fd1a-7bd2-4ad0-a593-09f1f5c06537-kube-api-access-4b2tf\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:37.575544 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:37.575513    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"
Oct 30 12:56:37.575544 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:37.575515    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq" event={"ID":"e227fd1a-7bd2-4ad0-a593-09f1f5c06537","Type":"ContainerDied","Data":"4685beb614e209614db8d1b36d82c4db8eec9b1004dbaa5f3f76d71985db410c"}
Oct 30 12:56:37.575754 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:37.575582    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4685beb614e209614db8d1b36d82c4db8eec9b1004dbaa5f3f76d71985db410c"
Oct 30 12:56:39.170152 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.170112    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq" podUID="c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed" containerName="payload-processing" containerID="cri-o://bc988f870ff9c26a2f5f249af80f146db0845d94289770ed868a8755e0b3de31" gracePeriod=25
Oct 30 12:56:39.205062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.205029    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf" podUID="67ad4195-3d80-4201-b3b9-fab610272e24" containerName="payload-pre-processing" containerID="cri-o://fb1e548b13fa5d6bdcd66018944a0915bcc141551039b7b5a49414f1f9b869d5" gracePeriod=25
Oct 30 12:56:39.533266 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.533241    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"
Oct 30 12:56:39.549253 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.549233    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"
Oct 30 12:56:39.618617 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.618588    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed-kube-api-access-42t96\" (UniqueName: \"kubernetes.io/projected/c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed-kube-api-access-42t96\") pod \"c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed\" (UID: \"c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed\") "
Oct 30 12:56:39.618883 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.618856    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ad4195-3d80-4201-b3b9-fab610272e24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ad4195-3d80-4201-b3b9-fab610272e24-plugins-config-volume\") pod \"67ad4195-3d80-4201-b3b9-fab610272e24\" (UID: \"67ad4195-3d80-4201-b3b9-fab610272e24\") "
Oct 30 12:56:39.619173 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.619156    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ad4195-3d80-4201-b3b9-fab610272e24-kube-api-access-n8gl5\" (UniqueName: \"kubernetes.io/projected/67ad4195-3d80-4201-b3b9-fab610272e24-kube-api-access-n8gl5\") pod \"67ad4195-3d80-4201-b3b9-fab610272e24\" (UID: \"67ad4195-3d80-4201-b3b9-fab610272e24\") "
Oct 30 12:56:39.619244 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.619230    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed-plugins-config-volume\") pod \"c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed\" (UID: \"c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed\") "
Oct 30 12:56:39.619835 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.619803    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ad4195-3d80-4201-b3b9-fab610272e24-plugins-config-volume" pod "67ad4195-3d80-4201-b3b9-fab610272e24" (UID: "67ad4195-3d80-4201-b3b9-fab610272e24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:56:39.620514 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.620489    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ad4195-3d80-4201-b3b9-fab610272e24-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:39.620866 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.620843    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed-plugins-config-volume" pod "c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed" (UID: "c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:56:39.623249 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.623217    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed-kube-api-access-42t96" pod "c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed" (UID: "c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed"). InnerVolumeSpecName "kube-api-access-42t96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:56:39.623546 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.623524    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ad4195-3d80-4201-b3b9-fab610272e24-kube-api-access-n8gl5" pod "67ad4195-3d80-4201-b3b9-fab610272e24" (UID: "67ad4195-3d80-4201-b3b9-fab610272e24"). InnerVolumeSpecName "kube-api-access-n8gl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:56:39.624430 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.624371    5837 generic.go:356] "Generic (PLEG): container finished" podID="67ad4195-3d80-4201-b3b9-fab610272e24" containerID="fb1e548b13fa5d6bdcd66018944a0915bcc141551039b7b5a49414f1f9b869d5" exitCode=0
Oct 30 12:56:39.624612 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.624592    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"
Oct 30 12:56:39.624676 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.624594    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf" event={"ID":"67ad4195-3d80-4201-b3b9-fab610272e24","Type":"ContainerDied","Data":"fb1e548b13fa5d6bdcd66018944a0915bcc141551039b7b5a49414f1f9b869d5"}
Oct 30 12:56:39.624728 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.624701    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf" event={"ID":"67ad4195-3d80-4201-b3b9-fab610272e24","Type":"ContainerDied","Data":"da8069d5125be695cfa66f1a444fb66f6f486ab6fe22c2e30318c478e1231cb4"}
Oct 30 12:56:39.624778 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.624739    5837 scope.go:122] "RemoveContainer" containerID="fb1e548b13fa5d6bdcd66018944a0915bcc141551039b7b5a49414f1f9b869d5"
Oct 30 12:56:39.629358 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.629329    5837 generic.go:356] "Generic (PLEG): container finished" podID="c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed" containerID="bc988f870ff9c26a2f5f249af80f146db0845d94289770ed868a8755e0b3de31" exitCode=0
Oct 30 12:56:39.629501 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.629413    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq" event={"ID":"c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed","Type":"ContainerDied","Data":"bc988f870ff9c26a2f5f249af80f146db0845d94289770ed868a8755e0b3de31"}
Oct 30 12:56:39.629501 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.629463    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq" event={"ID":"c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed","Type":"ContainerDied","Data":"7967a7d962be42653c9be2f866402dfeea9383ad12751e0e3a886af5f38117ea"}
Oct 30 12:56:39.629635 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.629515    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"
Oct 30 12:56:39.640703 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.640512    5837 scope.go:122] "RemoveContainer" containerID="fb1e548b13fa5d6bdcd66018944a0915bcc141551039b7b5a49414f1f9b869d5"
Oct 30 12:56:39.642749 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:56:39.641595    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1e548b13fa5d6bdcd66018944a0915bcc141551039b7b5a49414f1f9b869d5\": container with ID starting with fb1e548b13fa5d6bdcd66018944a0915bcc141551039b7b5a49414f1f9b869d5 not found: ID does not exist" containerID="fb1e548b13fa5d6bdcd66018944a0915bcc141551039b7b5a49414f1f9b869d5"
Oct 30 12:56:39.642749 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.641639    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1e548b13fa5d6bdcd66018944a0915bcc141551039b7b5a49414f1f9b869d5"} err="failed to get container status \"fb1e548b13fa5d6bdcd66018944a0915bcc141551039b7b5a49414f1f9b869d5\": rpc error: code = NotFound desc = could not find container \"fb1e548b13fa5d6bdcd66018944a0915bcc141551039b7b5a49414f1f9b869d5\": container with ID starting with fb1e548b13fa5d6bdcd66018944a0915bcc141551039b7b5a49414f1f9b869d5 not found: ID does not exist"
Oct 30 12:56:39.642749 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.641664    5837 scope.go:122] "RemoveContainer" containerID="bc988f870ff9c26a2f5f249af80f146db0845d94289770ed868a8755e0b3de31"
Oct 30 12:56:39.659040 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.659012    5837 scope.go:122] "RemoveContainer" containerID="bc988f870ff9c26a2f5f249af80f146db0845d94289770ed868a8755e0b3de31"
Oct 30 12:56:39.659336 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:56:39.659308    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc988f870ff9c26a2f5f249af80f146db0845d94289770ed868a8755e0b3de31\": container with ID starting with bc988f870ff9c26a2f5f249af80f146db0845d94289770ed868a8755e0b3de31 not found: ID does not exist" containerID="bc988f870ff9c26a2f5f249af80f146db0845d94289770ed868a8755e0b3de31"
Oct 30 12:56:39.659489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.659345    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc988f870ff9c26a2f5f249af80f146db0845d94289770ed868a8755e0b3de31"} err="failed to get container status \"bc988f870ff9c26a2f5f249af80f146db0845d94289770ed868a8755e0b3de31\": rpc error: code = NotFound desc = could not find container \"bc988f870ff9c26a2f5f249af80f146db0845d94289770ed868a8755e0b3de31\": container with ID starting with bc988f870ff9c26a2f5f249af80f146db0845d94289770ed868a8755e0b3de31 not found: ID does not exist"
Oct 30 12:56:39.692541 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.692476    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"]
Oct 30 12:56:39.697310 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.697287    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9e9fd-65498cb89d-tgdpq"]
Oct 30 12:56:39.721600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.721573    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42t96\" (UniqueName: \"kubernetes.io/projected/c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed-kube-api-access-42t96\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:39.721600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.721597    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8gl5\" (UniqueName: \"kubernetes.io/projected/67ad4195-3d80-4201-b3b9-fab610272e24-kube-api-access-n8gl5\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:39.721735 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.721608    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:56:39.727310 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.727288    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"]
Oct 30 12:56:39.730846 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:39.730826    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf"]
Oct 30 12:56:40.536089 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:40.536045    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9e9fd-59d57bf575-4ndlf" podUID="67ad4195-3d80-4201-b3b9-fab610272e24" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.116:50052\" within 1s: context deadline exceeded"
Oct 30 12:56:40.536536 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:56:40.536161    5837 logging.go:55] [core] [Channel #1712 SubChannel #1713]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.116:50052", ServerName: "10.134.0.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.116:50052: operation was canceled"
Oct 30 12:56:41.605695 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:41.605664    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ad4195-3d80-4201-b3b9-fab610272e24" path="/var/lib/kubelet/pods/67ad4195-3d80-4201-b3b9-fab610272e24/volumes"
Oct 30 12:56:41.606080 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:41.606065    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed" path="/var/lib/kubelet/pods/c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed/volumes"
Oct 30 12:56:45.094833 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:56:45.094792    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:56:51.836488 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:51.836457    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 12:56:51.837322 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:51.837302    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 12:56:53.029278 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:53.029247    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"]
Oct 30 12:56:53.040658 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:53.040634    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152b8586-d747j"]
Oct 30 12:56:53.607173 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:53.607131    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc947c90-b4bc-4fc0-9935-10706af098cd" path="/var/lib/kubelet/pods/bc947c90-b4bc-4fc0-9935-10706af098cd/volumes"
Oct 30 12:56:54.870073 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:54.870043    5837 generic.go:356] "Generic (PLEG): container finished" podID="def19b51-148c-4a5e-bdbe-313b17e37303" containerID="26613084cc223bc1a6f00fd2c431804fd022ed504917173b3be64ba42fa4a691" exitCode=0
Oct 30 12:56:54.870500 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:54.870118    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7" event={"ID":"def19b51-148c-4a5e-bdbe-313b17e37303","Type":"ContainerDied","Data":"26613084cc223bc1a6f00fd2c431804fd022ed504917173b3be64ba42fa4a691"}
Oct 30 12:56:54.870645 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:54.870628    5837 scope.go:122] "RemoveContainer" containerID="26613084cc223bc1a6f00fd2c431804fd022ed504917173b3be64ba42fa4a691"
Oct 30 12:56:54.872213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:54.872189    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:54.872280 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:54.872256    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:54.872317 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:54.872288    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:55.879678 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:55.879644    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7" event={"ID":"def19b51-148c-4a5e-bdbe-313b17e37303","Type":"ContainerStarted","Data":"7014b5cbe6f693d653dc2088bf802201fe559e1d657d105c5c51f916551ca070"}
Oct 30 12:56:56.844255 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:56.844224    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:56:56.845608 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:56.845585    5837 patch_prober.go:29] interesting pod/e2e-api-a-6bda55-openshift-default-757659f87-x82c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 12:56:56.845738 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:56.845635    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7" podUID="def19b51-148c-4a5e-bdbe-313b17e37303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 12:56:57.844880 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:57.844850    5837 patch_prober.go:29] interesting pod/e2e-api-a-6bda55-openshift-default-757659f87-x82c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 12:56:57.845246 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:57.844898    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7" podUID="def19b51-148c-4a5e-bdbe-313b17e37303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 12:56:58.843862 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:58.843832    5837 patch_prober.go:29] interesting pod/e2e-api-a-6bda55-openshift-default-757659f87-x82c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 12:56:58.844061 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:58.843885    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7" podUID="def19b51-148c-4a5e-bdbe-313b17e37303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 12:56:59.122103 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:59.122026    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:56:59.845419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:59.844826    5837 patch_prober.go:29] interesting pod/e2e-api-a-6bda55-openshift-default-757659f87-x82c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 12:56:59.845419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:56:59.844884    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7" podUID="def19b51-148c-4a5e-bdbe-313b17e37303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 12:57:00.467599 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.467549    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"]
Oct 30 12:57:00.468678 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.468568    5837 cpu_manager.go:414] "containerMap: removing container" podUID="67ad4195-3d80-4201-b3b9-fab610272e24" containerName="payload-pre-processing"
Oct 30 12:57:00.468678 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.468593    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ad4195-3d80-4201-b3b9-fab610272e24" containerName="payload-pre-processing"
Oct 30 12:57:00.468678 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.468616    5837 cpu_manager.go:414] "containerMap: removing container" podUID="e227fd1a-7bd2-4ad0-a593-09f1f5c06537" containerName="revoke-keys"
Oct 30 12:57:00.468678 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.468623    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e227fd1a-7bd2-4ad0-a593-09f1f5c06537" containerName="revoke-keys"
Oct 30 12:57:00.468678 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.468647    5837 cpu_manager.go:414] "containerMap: removing container" podUID="c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed" containerName="payload-processing"
Oct 30 12:57:00.468678 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.468656    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed" containerName="payload-processing"
Oct 30 12:57:00.469993 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.468864    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ad4195-3d80-4201-b3b9-fab610272e24" containerName="payload-pre-processing"
Oct 30 12:57:00.469993 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.468878    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e227fd1a-7bd2-4ad0-a593-09f1f5c06537" containerName="revoke-keys"
Oct 30 12:57:00.469993 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.468893    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e987bd-eb4c-4dc7-95fc-e3e053faa1ed" containerName="payload-processing"
Oct 30 12:57:00.469993 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.468913    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="484f0dd9-59d8-4690-a051-044c1fed4945" containerName="istio-proxy"
Oct 30 12:57:00.473246 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.473225    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.497766 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.495354    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-70621c-openshift-default-dockercfg-t4zd8\""
Oct 30 12:57:00.521631 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.521586    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"]
Oct 30 12:57:00.582273 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.582237    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-workload-socket\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.582463 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.582287    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-token\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.582463 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.582376    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-data\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.582601 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.582516    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istiod-ca-cert\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.582674 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.582658    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-podinfo\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.582738 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.582714    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r6t9\" (UniqueName: \"kubernetes.io/projected/e7c0f7fa-4a94-41ae-a846-8309e089d18e-kube-api-access-5r6t9\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.582795 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.582758    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-credential-socket\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.582856 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.582843    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-workload-certs\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.582913 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.582896    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-envoy\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.694420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.691645    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-data\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.694420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.691836    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istiod-ca-cert\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.694420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.691995    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-podinfo\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.694420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.692027    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r6t9\" (UniqueName: \"kubernetes.io/projected/e7c0f7fa-4a94-41ae-a846-8309e089d18e-kube-api-access-5r6t9\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.694420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.692105    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-credential-socket\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.694420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.692229    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-workload-certs\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.694420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.692284    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-envoy\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.694420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.692520    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-workload-socket\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.694420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.692551    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-token\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.694420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.693322    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-data\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.694420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.694082    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istiod-ca-cert\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.695127 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.694974    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-workload-certs\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.703688 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.695446    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-credential-socket\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.709421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.705082    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-workload-socket\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.732473 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.732021    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-podinfo\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.753753 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.753629    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-envoy\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.791029 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.790998    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-token\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.794633 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.794561    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r6t9\" (UniqueName: \"kubernetes.io/projected/e7c0f7fa-4a94-41ae-a846-8309e089d18e-kube-api-access-5r6t9\") pod \"e2e-api-b-70621c-openshift-default-74bc85477-kvzf4\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") " pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.821843 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.821806    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:00.847167 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.847138    5837 patch_prober.go:29] interesting pod/e2e-api-a-6bda55-openshift-default-757659f87-x82c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 12:57:00.847448 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:00.847418    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7" podUID="def19b51-148c-4a5e-bdbe-313b17e37303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 12:57:01.286751 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:57:01.286723    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c0f7fa_4a94_41ae_a846_8309e089d18e.slice/crio-5da72d611980f57d9d0331b7a2843910372b1677a48f0f1947cf4acc71cb1a06 WatchSource:0}: Error finding container 5da72d611980f57d9d0331b7a2843910372b1677a48f0f1947cf4acc71cb1a06: Status 404 returned error can't find the container with id 5da72d611980f57d9d0331b7a2843910372b1677a48f0f1947cf4acc71cb1a06
Oct 30 12:57:01.287257 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:01.287232    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"]
Oct 30 12:57:01.289374 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:01.289348    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:01.289481 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:01.289425    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:01.289481 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:01.289455    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:01.849329 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:01.849289    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:57:01.850652 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:01.850629    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:57:01.985280 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:01.985244    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4" event={"ID":"e7c0f7fa-4a94-41ae-a846-8309e089d18e","Type":"ContainerStarted","Data":"c26c377726944f54e9e85ae5869902e31227aef5e09b97fbe55dddb4cb24c7b4"}
Oct 30 12:57:01.985471 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:01.985300    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4" event={"ID":"e7c0f7fa-4a94-41ae-a846-8309e089d18e","Type":"ContainerStarted","Data":"5da72d611980f57d9d0331b7a2843910372b1677a48f0f1947cf4acc71cb1a06"}
Oct 30 12:57:02.028758 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:02.028697    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4" podStartSLOduration=2.0286781 podStartE2EDuration="2.0286781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:02.027179384 +0000 UTC m=+1210.551463602" watchObservedRunningTime="2026-10-30 12:57:02.0286781 +0000 UTC m=+1210.552962320"
Oct 30 12:57:02.241136 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:02.241063    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"]
Oct 30 12:57:02.248100 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:02.248031    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9e9fd-wnxzb"]
Oct 30 12:57:02.823051 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:02.823016    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:02.828480 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:02.828452    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:02.992902 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:02.992876    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:02.993876 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:02.993856    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:57:03.612447 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:03.612407    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f5c0a1c-62f7-4e90-967a-1c8461363273" path="/var/lib/kubelet/pods/0f5c0a1c-62f7-4e90-967a-1c8461363273/volumes"
Oct 30 12:57:06.305582 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.305551    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"]
Oct 30 12:57:06.309303 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.309282    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"
Oct 30 12:57:06.336580 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.336559    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-70621c\""
Oct 30 12:57:06.342034 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.342009    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"]
Oct 30 12:57:06.364035 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.364010    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"]
Oct 30 12:57:06.367572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.367551    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"
Oct 30 12:57:06.386332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.386299    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"]
Oct 30 12:57:06.417763 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.417724    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-70621c-dockercfg-4vrck\""
Oct 30 12:57:06.418822 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.418746    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcghs\" (UniqueName: \"kubernetes.io/projected/e9609568-7103-4f76-bbdc-877621e7487e-kube-api-access-zcghs\") pod \"payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt\" (UID: \"e9609568-7103-4f76-bbdc-877621e7487e\") " pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"
Oct 30 12:57:06.419131 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.418942    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9609568-7103-4f76-bbdc-877621e7487e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt\" (UID: \"e9609568-7103-4f76-bbdc-877621e7487e\") " pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"
Oct 30 12:57:06.534875 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.523372    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcghs\" (UniqueName: \"kubernetes.io/projected/e9609568-7103-4f76-bbdc-877621e7487e-kube-api-access-zcghs\") pod \"payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt\" (UID: \"e9609568-7103-4f76-bbdc-877621e7487e\") " pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"
Oct 30 12:57:06.534875 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.523624    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9609568-7103-4f76-bbdc-877621e7487e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt\" (UID: \"e9609568-7103-4f76-bbdc-877621e7487e\") " pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"
Oct 30 12:57:06.534875 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.523738    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e37dc1-7400-4517-9810-905a3cce65e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5\" (UID: \"d9e37dc1-7400-4517-9810-905a3cce65e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"
Oct 30 12:57:06.534875 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.523818    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nlp9\" (UniqueName: \"kubernetes.io/projected/d9e37dc1-7400-4517-9810-905a3cce65e2-kube-api-access-7nlp9\") pod \"payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5\" (UID: \"d9e37dc1-7400-4517-9810-905a3cce65e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"
Oct 30 12:57:06.534875 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.525348    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9609568-7103-4f76-bbdc-877621e7487e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt\" (UID: \"e9609568-7103-4f76-bbdc-877621e7487e\") " pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"
Oct 30 12:57:06.586473 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.584260    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcghs\" (UniqueName: \"kubernetes.io/projected/e9609568-7103-4f76-bbdc-877621e7487e-kube-api-access-zcghs\") pod \"payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt\" (UID: \"e9609568-7103-4f76-bbdc-877621e7487e\") " pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"
Oct 30 12:57:06.622937 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.622881    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"
Oct 30 12:57:06.625647 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.625620    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e37dc1-7400-4517-9810-905a3cce65e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5\" (UID: \"d9e37dc1-7400-4517-9810-905a3cce65e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"
Oct 30 12:57:06.628043 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.627636    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nlp9\" (UniqueName: \"kubernetes.io/projected/d9e37dc1-7400-4517-9810-905a3cce65e2-kube-api-access-7nlp9\") pod \"payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5\" (UID: \"d9e37dc1-7400-4517-9810-905a3cce65e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"
Oct 30 12:57:06.629220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.626963    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e37dc1-7400-4517-9810-905a3cce65e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5\" (UID: \"d9e37dc1-7400-4517-9810-905a3cce65e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"
Oct 30 12:57:06.647820 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.647788    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nlp9\" (UniqueName: \"kubernetes.io/projected/d9e37dc1-7400-4517-9810-905a3cce65e2-kube-api-access-7nlp9\") pod \"payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5\" (UID: \"d9e37dc1-7400-4517-9810-905a3cce65e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"
Oct 30 12:57:06.683135 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.683097    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"
Oct 30 12:57:06.942606 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:06.942559    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"]
Oct 30 12:57:06.942818 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:57:06.942700    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9609568_7103_4f76_bbdc_877621e7487e.slice/crio-68c3161627493b2a4ebaadd5cfb03194536e6a6e1cbac1717b3dc0838bbcc658 WatchSource:0}: Error finding container 68c3161627493b2a4ebaadd5cfb03194536e6a6e1cbac1717b3dc0838bbcc658: Status 404 returned error can't find the container with id 68c3161627493b2a4ebaadd5cfb03194536e6a6e1cbac1717b3dc0838bbcc658
Oct 30 12:57:07.025091 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:07.025062    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt" event={"ID":"e9609568-7103-4f76-bbdc-877621e7487e","Type":"ContainerStarted","Data":"68c3161627493b2a4ebaadd5cfb03194536e6a6e1cbac1717b3dc0838bbcc658"}
Oct 30 12:57:07.032384 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:57:07.032361    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e37dc1_7400_4517_9810_905a3cce65e2.slice/crio-deb52c1913fd835213ed9121e07c99a54cc6b7cc546a7e77f6c55fdcf49bd5e0 WatchSource:0}: Error finding container deb52c1913fd835213ed9121e07c99a54cc6b7cc546a7e77f6c55fdcf49bd5e0: Status 404 returned error can't find the container with id deb52c1913fd835213ed9121e07c99a54cc6b7cc546a7e77f6c55fdcf49bd5e0
Oct 30 12:57:07.032797 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:07.032777    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"]
Oct 30 12:57:08.039753 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:08.039713    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5" event={"ID":"d9e37dc1-7400-4517-9810-905a3cce65e2","Type":"ContainerStarted","Data":"f61587e901a4ed5a1b163a22c49c67d05fa3878e0ae08a5849785950c1650c56"}
Oct 30 12:57:08.040277 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:08.039798    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5" event={"ID":"d9e37dc1-7400-4517-9810-905a3cce65e2","Type":"ContainerStarted","Data":"deb52c1913fd835213ed9121e07c99a54cc6b7cc546a7e77f6c55fdcf49bd5e0"}
Oct 30 12:57:08.040568 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:08.040549    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"
Oct 30 12:57:08.045721 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:08.045691    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt" event={"ID":"e9609568-7103-4f76-bbdc-877621e7487e","Type":"ContainerStarted","Data":"18c9907ab7d5b68fc32173659abb34113688fa6bde329942bbe7fc1b799e6664"}
Oct 30 12:57:08.045864 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:08.045766    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"
Oct 30 12:57:08.081491 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:08.081434    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5" podStartSLOduration=2.081410356 podStartE2EDuration="2.081410356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:08.079382324 +0000 UTC m=+1216.603666544" watchObservedRunningTime="2026-10-30 12:57:08.081410356 +0000 UTC m=+1216.605694573"
Oct 30 12:57:08.120780 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:08.120725    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt" podStartSLOduration=2.12070997 podStartE2EDuration="2.12070997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:08.116655687 +0000 UTC m=+1216.640939905" watchObservedRunningTime="2026-10-30 12:57:08.12070997 +0000 UTC m=+1216.644994195"
Oct 30 12:57:08.242079 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:08.242044    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"]
Oct 30 12:57:08.247003 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:08.246977    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6bda55-f49zq"]
Oct 30 12:57:09.606961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:09.606928    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e227fd1a-7bd2-4ad0-a593-09f1f5c06537" path="/var/lib/kubelet/pods/e227fd1a-7bd2-4ad0-a593-09f1f5c06537/volumes"
Oct 30 12:57:12.990035 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:12.990002    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"]
Oct 30 12:57:12.995322 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:12.995300    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.016915 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.016887    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2bfb1b-openshift-default-dockercfg-tgvhs\""
Oct 30 12:57:13.036278 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.036243    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"]
Oct 30 12:57:13.094305 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.094268    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d739dd4-b062-4a32-a496-637ee6476b02-istio-token\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.094529 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.094386    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-istio-envoy\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.094529 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.094435    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-credential-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.094529 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.094461    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-istio-data\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.094529 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.094506    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d739dd4-b062-4a32-a496-637ee6476b02-istio-podinfo\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.094756 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.094593    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvcmg\" (UniqueName: \"kubernetes.io/projected/1d739dd4-b062-4a32-a496-637ee6476b02-kube-api-access-jvcmg\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.094756 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.094613    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-workload-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.094756 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.094629    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d739dd4-b062-4a32-a496-637ee6476b02-istiod-ca-cert\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.094756 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.094647    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-workload-certs\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.210420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.204414    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-istio-envoy\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.210420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.204472    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-credential-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.210420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.204511    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-istio-data\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.210420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.204575    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d739dd4-b062-4a32-a496-637ee6476b02-istio-podinfo\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.210420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.204702    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvcmg\" (UniqueName: \"kubernetes.io/projected/1d739dd4-b062-4a32-a496-637ee6476b02-kube-api-access-jvcmg\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.210420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.204731    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-workload-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.210420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.204759    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d739dd4-b062-4a32-a496-637ee6476b02-istiod-ca-cert\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.210420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.204785    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-workload-certs\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.210420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.204957    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d739dd4-b062-4a32-a496-637ee6476b02-istio-token\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.226697 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.226104    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-credential-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.226697 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.226354    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-istio-data\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.227999 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.227510    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-workload-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.227999 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.227759    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-workload-certs\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.228843 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.228821    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d739dd4-b062-4a32-a496-637ee6476b02-istiod-ca-cert\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.236459 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.230184    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d739dd4-b062-4a32-a496-637ee6476b02-istio-token\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.236869 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.236821    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-istio-envoy\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.246127 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.246067    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d739dd4-b062-4a32-a496-637ee6476b02-istio-podinfo\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.252669 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.252606    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvcmg\" (UniqueName: \"kubernetes.io/projected/1d739dd4-b062-4a32-a496-637ee6476b02-kube-api-access-jvcmg\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:13.314607 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:13.313681    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:14.598421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:14.595215    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"]
Oct 30 12:57:14.698455 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:57:14.623356    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d739dd4_b062_4a32_a496_637ee6476b02.slice/crio-7ce15977a337c20af6d3f66d963368d5dd5ae924496426691bb8c859c3ec77ec WatchSource:0}: Error finding container 7ce15977a337c20af6d3f66d963368d5dd5ae924496426691bb8c859c3ec77ec: Status 404 returned error can't find the container with id 7ce15977a337c20af6d3f66d963368d5dd5ae924496426691bb8c859c3ec77ec
Oct 30 12:57:14.728495 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:14.698781    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:14.728843 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:14.728806    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:14.729062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:14.729040    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:15.376748 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:15.376703    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:15.736505 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:15.735655    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"]
Oct 30 12:57:15.736505 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:15.735901    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7" podUID="def19b51-148c-4a5e-bdbe-313b17e37303" containerName="istio-proxy" containerID="cri-o://7014b5cbe6f693d653dc2088bf802201fe559e1d657d105c5c51f916551ca070" gracePeriod=30
Oct 30 12:57:15.777425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:15.759034    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:15.936748 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:15.851744    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:15.940418 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:15.937184    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" podUID="25644850-37a6-4820-8ed2-4f00fedd8274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.005484 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:57:15.994889    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:57:16.278607 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:16.278504    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm" event={"ID":"1d739dd4-b062-4a32-a496-637ee6476b02","Type":"ContainerStarted","Data":"7ce15977a337c20af6d3f66d963368d5dd5ae924496426691bb8c859c3ec77ec"}
Oct 30 12:57:16.389420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:16.384948    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.389420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:16.385841    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:57:16.749428 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:16.740569    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.801468 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:16.769990    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.801468 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:16.770779    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:57:16.929450 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:16.929415    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"]
Oct 30 12:57:17.095456 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:17.038652    5837 patch_prober.go:29] interesting pod/e2e-api-a-6bda55-openshift-default-757659f87-x82c7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:57:17.095456 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:17.038741    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7" podUID="def19b51-148c-4a5e-bdbe-313b17e37303" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:57:17.510477 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:17.486631    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:17.510477 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:17.486692    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk" podUID="f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:17.648572 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:17.648530    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:17.648843 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:17.648815    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4" podUID="f855b07d-e71c-491e-b430-0d33bdd77a5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:17.653414 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:17.648969    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:57:17.796623 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:17.796513    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:57:17.821239 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:17.805888    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:57:17.853591 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:17.853565    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"
Oct 30 12:57:17.918435 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:17.902708    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:57:17.935422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:17.925033    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:57:17.935689 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:17.931794    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:57:17.998180 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:17.959746    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:57:18.051475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:18.026005    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gh26\" (UniqueName: \"kubernetes.io/projected/a1df34c1-6cd6-437c-b22a-030354846012-kube-api-access-8gh26\") pod \"maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8\" (UID: \"a1df34c1-6cd6-437c-b22a-030354846012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"
Oct 30 12:57:18.051821 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:18.051787    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1df34c1-6cd6-437c-b22a-030354846012-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8\" (UID: \"a1df34c1-6cd6-437c-b22a-030354846012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"
Oct 30 12:57:18.052256 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:18.052231    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1df34c1-6cd6-437c-b22a-030354846012-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8\" (UID: \"a1df34c1-6cd6-437c-b22a-030354846012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"
Oct 30 12:57:18.173420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:18.155501    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1df34c1-6cd6-437c-b22a-030354846012-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8\" (UID: \"a1df34c1-6cd6-437c-b22a-030354846012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"
Oct 30 12:57:18.173835 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:18.173801    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gh26\" (UniqueName: \"kubernetes.io/projected/a1df34c1-6cd6-437c-b22a-030354846012-kube-api-access-8gh26\") pod \"maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8\" (UID: \"a1df34c1-6cd6-437c-b22a-030354846012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"
Oct 30 12:57:18.174145 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:18.174124    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1df34c1-6cd6-437c-b22a-030354846012-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8\" (UID: \"a1df34c1-6cd6-437c-b22a-030354846012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"
Oct 30 12:57:18.199748 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:18.168468    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1df34c1-6cd6-437c-b22a-030354846012-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8\" (UID: \"a1df34c1-6cd6-437c-b22a-030354846012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"
Oct 30 12:57:18.284784 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:18.284752    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"]
Oct 30 12:57:18.285072 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:18.285054    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:57:18.285202 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:18.285189    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"]
Oct 30 12:57:18.285320 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:18.285311    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e268819f-wqxls"]
Oct 30 12:57:18.543422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:18.519164    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gh26\" (UniqueName: \"kubernetes.io/projected/a1df34c1-6cd6-437c-b22a-030354846012-kube-api-access-8gh26\") pod \"maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8\" (UID: \"a1df34c1-6cd6-437c-b22a-030354846012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"
Oct 30 12:57:18.597429 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:18.562326    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1df34c1-6cd6-437c-b22a-030354846012-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8\" (UID: \"a1df34c1-6cd6-437c-b22a-030354846012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"
Oct 30 12:57:18.656687 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:18.656369    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:18.657321 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:18.657299    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"
Oct 30 12:57:19.061980 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:19.060814    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:57:19.063329 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:19.063279    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:57:19.197670 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:19.197638    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"
Oct 30 12:57:19.273176 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:19.273137    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm" event={"ID":"1d739dd4-b062-4a32-a496-637ee6476b02","Type":"ContainerStarted","Data":"4a055c20d83537e538b19f936d38c372844259b21dc554847af0f977d2e61d23"}
Oct 30 12:57:19.292428 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:19.288661    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"
Oct 30 12:57:19.333420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:19.320651    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:19.337513 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:19.337481    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"]
Oct 30 12:57:19.471909 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:19.471876    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:19.573479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:19.570048    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8478f5ef-kjw4q"]
Oct 30 12:57:19.620416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:19.576840    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:19.663427 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:19.659507    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:57:19.739585 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:19.739520    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm" podStartSLOduration=7.739501146 podStartE2EDuration="7.739501146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:19.353996916 +0000 UTC m=+1227.878281135" watchObservedRunningTime="2026-10-30 12:57:19.739501146 +0000 UTC m=+1228.263785363"
Oct 30 12:57:19.932479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:19.877796    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef9e74c-67c5-4fb6-ad94-5e9fe269792a" path="/var/lib/kubelet/pods/bef9e74c-67c5-4fb6-ad94-5e9fe269792a/volumes"
Oct 30 12:57:19.932479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:19.878663    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e" path="/var/lib/kubelet/pods/fe6dcb13-bdc9-4635-910b-1c5bed0a3b5e/volumes"
Oct 30 12:57:20.029647 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:20.029596    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:20.029980 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:20.029923    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" podUID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:20.083625 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:20.083561    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:20.084710 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:20.084677    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:20.459684 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:20.459655    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:20.472430 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:20.460867    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:20.495803 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:20.495767    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:57:21.064531 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:57:20.959271    5837 container.go:588] Failed to update stats for container "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef19b51_148c_4a5e_bdbe_313b17e37303.slice/crio-3bcdc4616f8c1a6ec21f9a3645f001f182edef01bdb22b58f7a128bbc5d0aaf1": error while statting cgroup v2: [read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef19b51_148c_4a5e_bdbe_313b17e37303.slice/crio-3bcdc4616f8c1a6ec21f9a3645f001f182edef01bdb22b58f7a128bbc5d0aaf1/memory.peak: no such device], continuing to push stats
Oct 30 12:57:21.129422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:21.092864    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.129422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:21.093751    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:57:21.184581 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:21.130262    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:21.184581 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:21.130343    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.490475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:21.484920    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.759349 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:21.752452    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:21.759349 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:21.752527    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:22.138495 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:22.094345    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:22.138495 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:22.094666    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:57:22.438015 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:22.437977    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"]
Oct 30 12:57:22.522744 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:22.504655    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:22.689416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:22.685759    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:22.689416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:22.685824    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:23.419928 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:23.419884    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:23.454422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:23.433537    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:57:23.527910 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:23.506948    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:23.575989 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:23.575943    5837 generic.go:356] "Generic (PLEG): container finished" podID="def19b51-148c-4a5e-bdbe-313b17e37303" containerID="7014b5cbe6f693d653dc2088bf802201fe559e1d657d105c5c51f916551ca070" exitCode=0
Oct 30 12:57:23.576511 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:23.576469    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7" event={"ID":"def19b51-148c-4a5e-bdbe-313b17e37303","Type":"ContainerDied","Data":"7014b5cbe6f693d653dc2088bf802201fe559e1d657d105c5c51f916551ca070"}
Oct 30 12:57:23.584367 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:23.576706    5837 scope.go:122] "RemoveContainer" containerID="26613084cc223bc1a6f00fd2c431804fd022ed504917173b3be64ba42fa4a691"
Oct 30 12:57:23.962566 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:23.946870    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:57:24.139923 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.139871    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def19b51-148c-4a5e-bdbe-313b17e37303-kube-api-access-mcl77\" (UniqueName: \"kubernetes.io/projected/def19b51-148c-4a5e-bdbe-313b17e37303-kube-api-access-mcl77\") pod \"def19b51-148c-4a5e-bdbe-313b17e37303\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") "
Oct 30 12:57:24.140220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.140179    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def19b51-148c-4a5e-bdbe-313b17e37303-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def19b51-148c-4a5e-bdbe-313b17e37303-istiod-ca-cert\") pod \"def19b51-148c-4a5e-bdbe-313b17e37303\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") "
Oct 30 12:57:24.140531 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.140513    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-workload-socket\") pod \"def19b51-148c-4a5e-bdbe-313b17e37303\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") "
Oct 30 12:57:24.140752 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.140714    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-credential-socket\") pod \"def19b51-148c-4a5e-bdbe-313b17e37303\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") "
Oct 30 12:57:24.141090 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.141072    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-istio-envoy\") pod \"def19b51-148c-4a5e-bdbe-313b17e37303\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") "
Oct 30 12:57:24.141279 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.141252    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-istio-data\") pod \"def19b51-148c-4a5e-bdbe-313b17e37303\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") "
Oct 30 12:57:24.141628 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.141611    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def19b51-148c-4a5e-bdbe-313b17e37303-istio-token\" (UniqueName: \"kubernetes.io/projected/def19b51-148c-4a5e-bdbe-313b17e37303-istio-token\") pod \"def19b51-148c-4a5e-bdbe-313b17e37303\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") "
Oct 30 12:57:24.141857 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.141840    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-workload-certs\") pod \"def19b51-148c-4a5e-bdbe-313b17e37303\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") "
Oct 30 12:57:24.217662 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.154115    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-workload-socket" pod "def19b51-148c-4a5e-bdbe-313b17e37303" (UID: "def19b51-148c-4a5e-bdbe-313b17e37303"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:24.217662 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.158484    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-credential-socket" pod "def19b51-148c-4a5e-bdbe-313b17e37303" (UID: "def19b51-148c-4a5e-bdbe-313b17e37303"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:24.217662 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.158863    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-workload-certs" pod "def19b51-148c-4a5e-bdbe-313b17e37303" (UID: "def19b51-148c-4a5e-bdbe-313b17e37303"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:24.217662 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.158911    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/def19b51-148c-4a5e-bdbe-313b17e37303-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def19b51-148c-4a5e-bdbe-313b17e37303-istio-podinfo\") pod \"def19b51-148c-4a5e-bdbe-313b17e37303\" (UID: \"def19b51-148c-4a5e-bdbe-313b17e37303\") "
Oct 30 12:57:24.217662 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.186090    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-istio-data" pod "def19b51-148c-4a5e-bdbe-313b17e37303" (UID: "def19b51-148c-4a5e-bdbe-313b17e37303"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:24.217662 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.186762    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:24.217662 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.186801    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:24.217662 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.186818    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:24.217662 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.186836    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:24.217662 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.215478    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def19b51-148c-4a5e-bdbe-313b17e37303-istiod-ca-cert" pod "def19b51-148c-4a5e-bdbe-313b17e37303" (UID: "def19b51-148c-4a5e-bdbe-313b17e37303"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:57:24.353594 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.315006    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def19b51-148c-4a5e-bdbe-313b17e37303-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:24.385179 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.383658    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def19b51-148c-4a5e-bdbe-313b17e37303-kube-api-access-mcl77" pod "def19b51-148c-4a5e-bdbe-313b17e37303" (UID: "def19b51-148c-4a5e-bdbe-313b17e37303"). InnerVolumeSpecName "kube-api-access-mcl77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:24.446425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.414798    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-istio-envoy" pod "def19b51-148c-4a5e-bdbe-313b17e37303" (UID: "def19b51-148c-4a5e-bdbe-313b17e37303"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:24.446425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.436500    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:24.446425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.438683    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def19b51-148c-4a5e-bdbe-313b17e37303-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:24.446425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.438713    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcl77\" (UniqueName: \"kubernetes.io/projected/def19b51-148c-4a5e-bdbe-313b17e37303-kube-api-access-mcl77\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:24.454415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.447227    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/def19b51-148c-4a5e-bdbe-313b17e37303-istio-podinfo" pod "def19b51-148c-4a5e-bdbe-313b17e37303" (UID: "def19b51-148c-4a5e-bdbe-313b17e37303"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:57:24.477344 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.461542    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:24.512460 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.500632    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:24.513741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.512831    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:57:24.571421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.561511    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def19b51-148c-4a5e-bdbe-313b17e37303-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:24.640783 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.640536    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def19b51-148c-4a5e-bdbe-313b17e37303-istio-token" pod "def19b51-148c-4a5e-bdbe-313b17e37303" (UID: "def19b51-148c-4a5e-bdbe-313b17e37303"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:24.720512 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:57:24.683767    5837 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.054s"
Oct 30 12:57:24.720512 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.683838    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:57:24.720512 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.683878    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8" event={"ID":"a1df34c1-6cd6-437c-b22a-030354846012","Type":"ContainerStarted","Data":"e85c372d24937cc8c3e55eaa4d3f8ed2f623eeae28bd2520d1249d00b2c29c0a"}
Oct 30 12:57:24.733430 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.725926    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:57:24.733430 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.728471    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:57:24.734654 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.734441    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def19b51-148c-4a5e-bdbe-313b17e37303-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:24.891275 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.891241    5837 generic.go:356] "Generic (PLEG): container finished" podID="a1df34c1-6cd6-437c-b22a-030354846012" containerID="256da51b98f9c2df20b146c072269612a8df08b28524461a25c18c840a6ca161" exitCode=0
Oct 30 12:57:24.892408 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.891831    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8" event={"ID":"a1df34c1-6cd6-437c-b22a-030354846012","Type":"ContainerDied","Data":"256da51b98f9c2df20b146c072269612a8df08b28524461a25c18c840a6ca161"}
Oct 30 12:57:24.894945 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.894915    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7" event={"ID":"def19b51-148c-4a5e-bdbe-313b17e37303","Type":"ContainerDied","Data":"3bcdc4616f8c1a6ec21f9a3645f001f182edef01bdb22b58f7a128bbc5d0aaf1"}
Oct 30 12:57:24.896893 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.896872    5837 scope.go:122] "RemoveContainer" containerID="7014b5cbe6f693d653dc2088bf802201fe559e1d657d105c5c51f916551ca070"
Oct 30 12:57:24.897106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:24.894672    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"
Oct 30 12:57:25.031262 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.031165    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"]
Oct 30 12:57:25.038571 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.038545    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6bda55-openshift-default-757659f87-x82c7"]
Oct 30 12:57:25.121664 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.121535    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"]
Oct 30 12:57:25.124627 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.124583    5837 cpu_manager.go:414] "containerMap: removing container" podUID="def19b51-148c-4a5e-bdbe-313b17e37303" containerName="istio-proxy"
Oct 30 12:57:25.124627 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.124629    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def19b51-148c-4a5e-bdbe-313b17e37303" containerName="istio-proxy"
Oct 30 12:57:25.124863 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.124659    5837 cpu_manager.go:414] "containerMap: removing container" podUID="def19b51-148c-4a5e-bdbe-313b17e37303" containerName="istio-proxy"
Oct 30 12:57:25.124863 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.124686    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def19b51-148c-4a5e-bdbe-313b17e37303" containerName="istio-proxy"
Oct 30 12:57:25.125052 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.125035    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="def19b51-148c-4a5e-bdbe-313b17e37303" containerName="istio-proxy"
Oct 30 12:57:25.125121 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.125060    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="def19b51-148c-4a5e-bdbe-313b17e37303" containerName="istio-proxy"
Oct 30 12:57:25.137752 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.132652    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"]
Oct 30 12:57:25.137752 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.132801    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"
Oct 30 12:57:25.228442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.189100    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2bfb1b-dockercfg-k7lm2\""
Oct 30 12:57:25.229228 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.201662    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2bfb1b\""
Oct 30 12:57:25.287282 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.266477    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229\" (UID: \"8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"
Oct 30 12:57:25.298424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.287744    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq9vg\" (UniqueName: \"kubernetes.io/projected/8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b-kube-api-access-nq9vg\") pod \"payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229\" (UID: \"8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"
Oct 30 12:57:25.450444 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.448651    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229\" (UID: \"8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"
Oct 30 12:57:25.450444 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.449208    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq9vg\" (UniqueName: \"kubernetes.io/projected/8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b-kube-api-access-nq9vg\") pod \"payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229\" (UID: \"8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"
Oct 30 12:57:25.450444 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.449925    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229\" (UID: \"8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"
Oct 30 12:57:25.472132 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.472095    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq9vg\" (UniqueName: \"kubernetes.io/projected/8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b-kube-api-access-nq9vg\") pod \"payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229\" (UID: \"8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"
Oct 30 12:57:25.578450 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.578362    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"
Oct 30 12:57:25.619634 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.619592    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def19b51-148c-4a5e-bdbe-313b17e37303" path="/var/lib/kubelet/pods/def19b51-148c-4a5e-bdbe-313b17e37303/volumes"
Oct 30 12:57:25.929484 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:25.929456    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"]
Oct 30 12:57:26.157554 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:26.157528    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"
Oct 30 12:57:26.261664 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:26.261630    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1df34c1-6cd6-437c-b22a-030354846012-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1df34c1-6cd6-437c-b22a-030354846012-maas-api-openshift-ca\") pod \"a1df34c1-6cd6-437c-b22a-030354846012\" (UID: \"a1df34c1-6cd6-437c-b22a-030354846012\") "
Oct 30 12:57:26.262220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:26.261707    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1df34c1-6cd6-437c-b22a-030354846012-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1df34c1-6cd6-437c-b22a-030354846012-maas-api-xks-ca\") pod \"a1df34c1-6cd6-437c-b22a-030354846012\" (UID: \"a1df34c1-6cd6-437c-b22a-030354846012\") "
Oct 30 12:57:26.262220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:26.261785    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1df34c1-6cd6-437c-b22a-030354846012-kube-api-access-8gh26\" (UniqueName: \"kubernetes.io/projected/a1df34c1-6cd6-437c-b22a-030354846012-kube-api-access-8gh26\") pod \"a1df34c1-6cd6-437c-b22a-030354846012\" (UID: \"a1df34c1-6cd6-437c-b22a-030354846012\") "
Oct 30 12:57:26.262220 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:26.261962    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1df34c1-6cd6-437c-b22a-030354846012-maas-api-openshift-ca" pod "a1df34c1-6cd6-437c-b22a-030354846012" (UID: "a1df34c1-6cd6-437c-b22a-030354846012"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:57:26.262445 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:26.262285    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1df34c1-6cd6-437c-b22a-030354846012-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:26.264133 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:26.264107    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1df34c1-6cd6-437c-b22a-030354846012-maas-api-xks-ca" pod "a1df34c1-6cd6-437c-b22a-030354846012" (UID: "a1df34c1-6cd6-437c-b22a-030354846012"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:57:26.264541 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:26.264518    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1df34c1-6cd6-437c-b22a-030354846012-kube-api-access-8gh26" pod "a1df34c1-6cd6-437c-b22a-030354846012" (UID: "a1df34c1-6cd6-437c-b22a-030354846012"). InnerVolumeSpecName "kube-api-access-8gh26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:26.363773 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:26.363734    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1df34c1-6cd6-437c-b22a-030354846012-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:26.363773 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:26.363768    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gh26\" (UniqueName: \"kubernetes.io/projected/a1df34c1-6cd6-437c-b22a-030354846012-kube-api-access-8gh26\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:26.996416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:26.988386    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229" event={"ID":"8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b","Type":"ContainerStarted","Data":"40d5564404d6db60592f79c17f5a01047f97539d472f58b71a15e86ed00782c3"}
Oct 30 12:57:26.996416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:26.988497    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229" event={"ID":"8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b","Type":"ContainerStarted","Data":"966a7c5afc4389be380bb8ca694242ab168384e33beaaebfa1b42b10af3e935a"}
Oct 30 12:57:26.996416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:26.991461    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"
Oct 30 12:57:27.006486 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.006457    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8" event={"ID":"a1df34c1-6cd6-437c-b22a-030354846012","Type":"ContainerDied","Data":"e85c372d24937cc8c3e55eaa4d3f8ed2f623eeae28bd2520d1249d00b2c29c0a"}
Oct 30 12:57:27.006928 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.006910    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e85c372d24937cc8c3e55eaa4d3f8ed2f623eeae28bd2520d1249d00b2c29c0a"
Oct 30 12:57:27.007075 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.006850    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"
Oct 30 12:57:27.031955 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.031885    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229" podStartSLOduration=2.031867961 podStartE2EDuration="2.031867961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:27.03083927 +0000 UTC m=+1235.555123500" watchObservedRunningTime="2026-10-30 12:57:27.031867961 +0000 UTC m=+1235.556152180"
Oct 30 12:57:27.509972 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.509932    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"]
Oct 30 12:57:27.513660 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.513067    5837 cpu_manager.go:414] "containerMap: removing container" podUID="a1df34c1-6cd6-437c-b22a-030354846012" containerName="revoke-keys"
Oct 30 12:57:27.513660 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.513095    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1df34c1-6cd6-437c-b22a-030354846012" containerName="revoke-keys"
Oct 30 12:57:27.513660 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.513305    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1df34c1-6cd6-437c-b22a-030354846012" containerName="revoke-keys"
Oct 30 12:57:27.517880 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.517858    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.524839 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.524810    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"]
Oct 30 12:57:27.531991 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.531967    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6a4499-openshift-default-dockercfg-2jtjs\""
Oct 30 12:57:27.610304 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.610269    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-istio-data\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.610612 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.610589    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-workload-socket\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.610805 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.610787    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0312c8bf-b31d-4923-b089-13a6558da515-istio-podinfo\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.610967 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.610951    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0312c8bf-b31d-4923-b089-13a6558da515-istio-token\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.611091 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.611076    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0312c8bf-b31d-4923-b089-13a6558da515-istiod-ca-cert\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.611203 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.611186    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b665p\" (UniqueName: \"kubernetes.io/projected/0312c8bf-b31d-4923-b089-13a6558da515-kube-api-access-b665p\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.611314 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.611300    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-istio-envoy\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.611483 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.611465    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-workload-certs\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.611717 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.611700    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-credential-socket\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.713796 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.713756    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0312c8bf-b31d-4923-b089-13a6558da515-istiod-ca-cert\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.713980 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.713815    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b665p\" (UniqueName: \"kubernetes.io/projected/0312c8bf-b31d-4923-b089-13a6558da515-kube-api-access-b665p\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.713980 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.713851    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-istio-envoy\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.714956 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.713955    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-workload-certs\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.714956 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.714479    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-credential-socket\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.714956 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.714498    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-workload-certs\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.714956 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.714583    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-istio-data\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.714956 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.714589    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0312c8bf-b31d-4923-b089-13a6558da515-istiod-ca-cert\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.714956 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.714708    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-workload-socket\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.714956 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.714892    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-istio-data\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.714956 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.714900    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0312c8bf-b31d-4923-b089-13a6558da515-istio-podinfo\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.714956 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.714965    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-credential-socket\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.715535 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.714997    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0312c8bf-b31d-4923-b089-13a6558da515-istio-token\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.715535 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.715015    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-workload-socket\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.717383 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.717360    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-istio-envoy\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.717546 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.717526    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0312c8bf-b31d-4923-b089-13a6558da515-istio-podinfo\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.733574 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.733552    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0312c8bf-b31d-4923-b089-13a6558da515-istio-token\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.734035 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.734011    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b665p\" (UniqueName: \"kubernetes.io/projected/0312c8bf-b31d-4923-b089-13a6558da515-kube-api-access-b665p\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:27.837594 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:27.837552    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:28.165922 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:57:28.165890    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0312c8bf_b31d_4923_b089_13a6558da515.slice/crio-4d3f74f32dbb15bc8848e4c50ce1f056fd86cf69ca8854318ec7a642b0f428fe WatchSource:0}: Error finding container 4d3f74f32dbb15bc8848e4c50ce1f056fd86cf69ca8854318ec7a642b0f428fe: Status 404 returned error can't find the container with id 4d3f74f32dbb15bc8848e4c50ce1f056fd86cf69ca8854318ec7a642b0f428fe
Oct 30 12:57:28.168563 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:28.168536    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"]
Oct 30 12:57:28.169874 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:28.169837    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:28.169977 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:28.169917    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:28.170068 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:28.169980    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:29.028091 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:29.028035    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6" event={"ID":"0312c8bf-b31d-4923-b089-13a6558da515","Type":"ContainerStarted","Data":"ee8206102c7640551e5561db1048a21a34455ed02faf3b937b9cf80f381c56dd"}
Oct 30 12:57:29.028280 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:29.028145    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6" event={"ID":"0312c8bf-b31d-4923-b089-13a6558da515","Type":"ContainerStarted","Data":"4d3f74f32dbb15bc8848e4c50ce1f056fd86cf69ca8854318ec7a642b0f428fe"}
Oct 30 12:57:29.115329 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:29.115255    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6" podStartSLOduration=2.115234569 podStartE2EDuration="2.115234569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:29.079268009 +0000 UTC m=+1237.603552241" watchObservedRunningTime="2026-10-30 12:57:29.115234569 +0000 UTC m=+1237.639518790"
Oct 30 12:57:29.853553 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:29.846764    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:29.858426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:29.857539    5837 patch_prober.go:29] interesting pod/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 12:57:29.858426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:29.857607    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6" podUID="0312c8bf-b31d-4923-b089-13a6558da515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 12:57:30.839143 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:30.839106    5837 patch_prober.go:29] interesting pod/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 12:57:30.839331 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:30.839172    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6" podUID="0312c8bf-b31d-4923-b089-13a6558da515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 12:57:31.644108 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.644065    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"]
Oct 30 12:57:31.710874 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.710710    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"]
Oct 30 12:57:31.711175 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.710818    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"
Oct 30 12:57:31.730438 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.730373    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:57:31.741729 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.741634    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:57:31.741729 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.741658    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:57:31.741729 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.741706    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:57:31.832431 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.832378    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw\" (UID: \"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"
Oct 30 12:57:31.832601 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.832467    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c498b\" (UniqueName: \"kubernetes.io/projected/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-kube-api-access-c498b\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw\" (UID: \"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"
Oct 30 12:57:31.832601 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.832508    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw\" (UID: \"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"
Oct 30 12:57:31.838516 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.838494    5837 patch_prober.go:29] interesting pod/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 12:57:31.838636 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.838545    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6" podUID="0312c8bf-b31d-4923-b089-13a6558da515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 12:57:31.933970 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.933887    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw\" (UID: \"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"
Oct 30 12:57:31.933970 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.933931    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c498b\" (UniqueName: \"kubernetes.io/projected/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-kube-api-access-c498b\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw\" (UID: \"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"
Oct 30 12:57:31.934189 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.933988    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw\" (UID: \"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"
Oct 30 12:57:31.935118 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.935094    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw\" (UID: \"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"
Oct 30 12:57:31.937018 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.936996    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw\" (UID: \"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"
Oct 30 12:57:31.953802 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:31.953776    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c498b\" (UniqueName: \"kubernetes.io/projected/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-kube-api-access-c498b\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw\" (UID: \"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"
Oct 30 12:57:32.047951 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:32.047919    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"
Oct 30 12:57:32.532856 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:32.530545    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"]
Oct 30 12:57:32.534805 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:57:32.534674    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f2757f_0c3e_4cdf_97a8_5eb37649adc0.slice/crio-5a52ab15bc3fd33d2e7464bcfb0328a1811823f80910a91eca0a40ede6270562 WatchSource:0}: Error finding container 5a52ab15bc3fd33d2e7464bcfb0328a1811823f80910a91eca0a40ede6270562: Status 404 returned error can't find the container with id 5a52ab15bc3fd33d2e7464bcfb0328a1811823f80910a91eca0a40ede6270562
Oct 30 12:57:32.838584 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:32.838544    5837 patch_prober.go:29] interesting pod/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 12:57:32.839005 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:32.838618    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6" podUID="0312c8bf-b31d-4923-b089-13a6558da515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 12:57:33.065180 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:33.065153    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw" event={"ID":"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0","Type":"ContainerStarted","Data":"d0b406065c330d5714b9226078467123796654a73d412ce7c231d4360b225f2c"}
Oct 30 12:57:33.065362 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:33.065211    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw" event={"ID":"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0","Type":"ContainerStarted","Data":"5a52ab15bc3fd33d2e7464bcfb0328a1811823f80910a91eca0a40ede6270562"}
Oct 30 12:57:33.378540 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:33.378508    5837 patch_prober.go:29] interesting pod/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:33.380104 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:33.378556    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" podUID="467baab9-bc62-4c3f-a034-651a6491271c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:33.838635 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:33.838600    5837 patch_prober.go:29] interesting pod/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 12:57:33.839043 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:33.838656    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6" podUID="0312c8bf-b31d-4923-b089-13a6558da515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 12:57:34.123787 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:34.123428    5837 generic.go:356] "Generic (PLEG): container finished" podID="467baab9-bc62-4c3f-a034-651a6491271c" containerID="6a55cad6a1616762cea51709c2c8eb27400cd6310ee82b191295227aa036ccea" exitCode=0
Oct 30 12:57:34.123787 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:34.123591    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" event={"ID":"467baab9-bc62-4c3f-a034-651a6491271c","Type":"ContainerDied","Data":"6a55cad6a1616762cea51709c2c8eb27400cd6310ee82b191295227aa036ccea"}
Oct 30 12:57:34.127609 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:34.124221    5837 scope.go:122] "RemoveContainer" containerID="6a55cad6a1616762cea51709c2c8eb27400cd6310ee82b191295227aa036ccea"
Oct 30 12:57:34.139163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:34.134827    5837 generic.go:356] "Generic (PLEG): container finished" podID="b1f2757f-0c3e-4cdf-97a8-5eb37649adc0" containerID="d0b406065c330d5714b9226078467123796654a73d412ce7c231d4360b225f2c" exitCode=0
Oct 30 12:57:34.139163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:34.134877    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw" event={"ID":"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0","Type":"ContainerDied","Data":"d0b406065c330d5714b9226078467123796654a73d412ce7c231d4360b225f2c"}
Oct 30 12:57:34.156415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:34.145030    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:34.156415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:34.145132    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:34.156415 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:34.145181    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:34.843095 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:34.843067    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:34.843558 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:34.843518    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:34.844764 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:34.844743    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:57:35.157562 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:35.157435    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" event={"ID":"467baab9-bc62-4c3f-a034-651a6491271c","Type":"ContainerStarted","Data":"ca7c09494793fb68faf1add666af94531eb1acb5d3fe26e3e24add7eb5c15f92"}
Oct 30 12:57:35.371539 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:35.371515    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"
Oct 30 12:57:35.438516 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:35.438438    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-maas-api-xks-ca\") pod \"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0\" (UID: \"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0\") "
Oct 30 12:57:35.438657 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:35.438640    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-maas-api-openshift-ca\") pod \"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0\" (UID: \"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0\") "
Oct 30 12:57:35.438726 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:35.438711    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-kube-api-access-c498b\" (UniqueName: \"kubernetes.io/projected/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-kube-api-access-c498b\") pod \"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0\" (UID: \"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0\") "
Oct 30 12:57:35.439082 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:35.439049    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-maas-api-openshift-ca" pod "b1f2757f-0c3e-4cdf-97a8-5eb37649adc0" (UID: "b1f2757f-0c3e-4cdf-97a8-5eb37649adc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:57:35.439533 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:35.439509    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:35.440909 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:35.440878    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-maas-api-xks-ca" pod "b1f2757f-0c3e-4cdf-97a8-5eb37649adc0" (UID: "b1f2757f-0c3e-4cdf-97a8-5eb37649adc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:57:35.441382 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:35.441363    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-kube-api-access-c498b" pod "b1f2757f-0c3e-4cdf-97a8-5eb37649adc0" (UID: "b1f2757f-0c3e-4cdf-97a8-5eb37649adc0"). InnerVolumeSpecName "kube-api-access-c498b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:35.540847 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:35.540809    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:35.540847 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:35.540845    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c498b\" (UniqueName: \"kubernetes.io/projected/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0-kube-api-access-c498b\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:35.759766 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:35.759727    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:57:35.759958 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:35.759853    5837 patch_prober.go:29] interesting pod/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 12:57:35.759958 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:35.759930    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" podUID="467baab9-bc62-4c3f-a034-651a6491271c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 12:57:36.166277 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:36.166193    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"
Oct 30 12:57:36.166277 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:36.166231    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw" event={"ID":"b1f2757f-0c3e-4cdf-97a8-5eb37649adc0","Type":"ContainerDied","Data":"5a52ab15bc3fd33d2e7464bcfb0328a1811823f80910a91eca0a40ede6270562"}
Oct 30 12:57:36.166277 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:36.166280    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a52ab15bc3fd33d2e7464bcfb0328a1811823f80910a91eca0a40ede6270562"
Oct 30 12:57:36.759962 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:36.759931    5837 patch_prober.go:29] interesting pod/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 12:57:36.760182 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:36.759999    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" podUID="467baab9-bc62-4c3f-a034-651a6491271c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 12:57:37.760747 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:37.760674    5837 patch_prober.go:29] interesting pod/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 12:57:37.761482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:37.761431    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" podUID="467baab9-bc62-4c3f-a034-651a6491271c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 12:57:38.028445 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:38.028345    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"]
Oct 30 12:57:38.039382 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:38.039351    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3420d37c-hk2fs"]
Oct 30 12:57:38.760046 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:38.760012    5837 patch_prober.go:29] interesting pod/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 12:57:38.760302 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:38.760073    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" podUID="467baab9-bc62-4c3f-a034-651a6491271c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 12:57:39.031361 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:39.031288    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"
Oct 30 12:57:39.618163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:39.618132    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347d1e37-c062-4aa0-9a35-60bc76148c83" path="/var/lib/kubelet/pods/347d1e37-c062-4aa0-9a35-60bc76148c83/volumes"
Oct 30 12:57:39.759823 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:39.759788    5837 patch_prober.go:29] interesting pod/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 12:57:39.760033 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:39.759856    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" podUID="467baab9-bc62-4c3f-a034-651a6491271c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 12:57:40.760191 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:40.760152    5837 patch_prober.go:29] interesting pod/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 12:57:40.760661 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:40.760227    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" podUID="467baab9-bc62-4c3f-a034-651a6491271c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 12:57:41.764139 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:41.764100    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:57:41.764667 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:41.764649    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:57:41.765287 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:41.765267    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:57:42.034021 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:42.033942    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"]
Oct 30 12:57:42.038386 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:42.038360    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0100fcb4-z59nz"]
Oct 30 12:57:43.101421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:43.085350    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"]
Oct 30 12:57:43.114970 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:43.114923    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555fdd9c-kxwqb"]
Oct 30 12:57:43.606277 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:43.606244    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39492b3b-a3d5-4456-a570-1cd0a9677c2b" path="/var/lib/kubelet/pods/39492b3b-a3d5-4456-a570-1cd0a9677c2b/volumes"
Oct 30 12:57:43.606749 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:43.606735    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c040266d-cb76-46cf-89f9-4ac58d484f14" path="/var/lib/kubelet/pods/c040266d-cb76-46cf-89f9-4ac58d484f14/volumes"
Oct 30 12:57:45.095543 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:57:45.095508    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:57:48.010762 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.010726    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"]
Oct 30 12:57:48.011658 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.011636    5837 cpu_manager.go:414] "containerMap: removing container" podUID="b1f2757f-0c3e-4cdf-97a8-5eb37649adc0" containerName="revoke-keys"
Oct 30 12:57:48.011658 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.011657    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f2757f-0c3e-4cdf-97a8-5eb37649adc0" containerName="revoke-keys"
Oct 30 12:57:48.011842 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.011828    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f2757f-0c3e-4cdf-97a8-5eb37649adc0" containerName="revoke-keys"
Oct 30 12:57:48.016615 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.016591    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"
Oct 30 12:57:48.053808 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.048717    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:57:48.053808 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.052282    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:57:48.053808 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.052423    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:57:48.081561 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.070933    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:57:48.182961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.166770    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2983f91d-27e6-4289-80c6-0a5bb4162128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b14f5479-ncjks\" (UID: \"2983f91d-27e6-4289-80c6-0a5bb4162128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"
Oct 30 12:57:48.182961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.166928    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2983f91d-27e6-4289-80c6-0a5bb4162128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b14f5479-ncjks\" (UID: \"2983f91d-27e6-4289-80c6-0a5bb4162128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"
Oct 30 12:57:48.182961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.167089    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5tcj\" (UniqueName: \"kubernetes.io/projected/2983f91d-27e6-4289-80c6-0a5bb4162128-kube-api-access-r5tcj\") pod \"maas-api-revoke-sub-b14f5479-ncjks\" (UID: \"2983f91d-27e6-4289-80c6-0a5bb4162128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"
Oct 30 12:57:48.182961 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.182843    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"]
Oct 30 12:57:48.268331 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.268239    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5tcj\" (UniqueName: \"kubernetes.io/projected/2983f91d-27e6-4289-80c6-0a5bb4162128-kube-api-access-r5tcj\") pod \"maas-api-revoke-sub-b14f5479-ncjks\" (UID: \"2983f91d-27e6-4289-80c6-0a5bb4162128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"
Oct 30 12:57:48.268555 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.268503    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2983f91d-27e6-4289-80c6-0a5bb4162128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b14f5479-ncjks\" (UID: \"2983f91d-27e6-4289-80c6-0a5bb4162128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"
Oct 30 12:57:48.268682 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.268629    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2983f91d-27e6-4289-80c6-0a5bb4162128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b14f5479-ncjks\" (UID: \"2983f91d-27e6-4289-80c6-0a5bb4162128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"
Oct 30 12:57:48.269280 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.269260    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2983f91d-27e6-4289-80c6-0a5bb4162128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b14f5479-ncjks\" (UID: \"2983f91d-27e6-4289-80c6-0a5bb4162128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"
Oct 30 12:57:48.271216 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.271183    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2983f91d-27e6-4289-80c6-0a5bb4162128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b14f5479-ncjks\" (UID: \"2983f91d-27e6-4289-80c6-0a5bb4162128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"
Oct 30 12:57:48.287506 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.287475    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5tcj\" (UniqueName: \"kubernetes.io/projected/2983f91d-27e6-4289-80c6-0a5bb4162128-kube-api-access-r5tcj\") pod \"maas-api-revoke-sub-b14f5479-ncjks\" (UID: \"2983f91d-27e6-4289-80c6-0a5bb4162128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"
Oct 30 12:57:48.460485 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.460452    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"
Oct 30 12:57:48.764868 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:48.764831    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"]
Oct 30 12:57:48.765252 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:57:48.765224    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2983f91d_27e6_4289_80c6_0a5bb4162128.slice/crio-e18ca7f156bf7044133cc1272a72ed3996c10e6fdb6ae7b43b21be7c69af785e WatchSource:0}: Error finding container e18ca7f156bf7044133cc1272a72ed3996c10e6fdb6ae7b43b21be7c69af785e: Status 404 returned error can't find the container with id e18ca7f156bf7044133cc1272a72ed3996c10e6fdb6ae7b43b21be7c69af785e
Oct 30 12:57:49.337704 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:49.337657    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks" event={"ID":"2983f91d-27e6-4289-80c6-0a5bb4162128","Type":"ContainerStarted","Data":"ab04d4e2c580929e587a093ae9c6ebf55228db9baed18853997fcfa7ec82080f"}
Oct 30 12:57:49.338195 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:49.337719    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks" event={"ID":"2983f91d-27e6-4289-80c6-0a5bb4162128","Type":"ContainerStarted","Data":"e18ca7f156bf7044133cc1272a72ed3996c10e6fdb6ae7b43b21be7c69af785e"}
Oct 30 12:57:49.375715 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:49.375587    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks" podStartSLOduration=2.375573311 podStartE2EDuration="2.375573311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:49.373776295 +0000 UTC m=+1257.898060514" watchObservedRunningTime="2026-10-30 12:57:49.375573311 +0000 UTC m=+1257.899857530"
Oct 30 12:57:50.347848 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:50.347812    5837 generic.go:356] "Generic (PLEG): container finished" podID="2983f91d-27e6-4289-80c6-0a5bb4162128" containerID="ab04d4e2c580929e587a093ae9c6ebf55228db9baed18853997fcfa7ec82080f" exitCode=0
Oct 30 12:57:50.347848 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:50.347849    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks" event={"ID":"2983f91d-27e6-4289-80c6-0a5bb4162128","Type":"ContainerDied","Data":"ab04d4e2c580929e587a093ae9c6ebf55228db9baed18853997fcfa7ec82080f"}
Oct 30 12:57:51.325670 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.325643    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"]
Oct 30 12:57:51.330238 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.330218    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"
Oct 30 12:57:51.360010 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.359929    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"]
Oct 30 12:57:51.415282 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.415237    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/456435f5-064d-48e8-8b14-855cf23581c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx\" (UID: \"456435f5-064d-48e8-8b14-855cf23581c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"
Oct 30 12:57:51.415496 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.415292    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4p4k\" (UniqueName: \"kubernetes.io/projected/456435f5-064d-48e8-8b14-855cf23581c7-kube-api-access-v4p4k\") pod \"maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx\" (UID: \"456435f5-064d-48e8-8b14-855cf23581c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"
Oct 30 12:57:51.415496 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.415389    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/456435f5-064d-48e8-8b14-855cf23581c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx\" (UID: \"456435f5-064d-48e8-8b14-855cf23581c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"
Oct 30 12:57:51.516804 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.516674    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/456435f5-064d-48e8-8b14-855cf23581c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx\" (UID: \"456435f5-064d-48e8-8b14-855cf23581c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"
Oct 30 12:57:51.516804 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.516710    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4p4k\" (UniqueName: \"kubernetes.io/projected/456435f5-064d-48e8-8b14-855cf23581c7-kube-api-access-v4p4k\") pod \"maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx\" (UID: \"456435f5-064d-48e8-8b14-855cf23581c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"
Oct 30 12:57:51.516804 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.516739    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/456435f5-064d-48e8-8b14-855cf23581c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx\" (UID: \"456435f5-064d-48e8-8b14-855cf23581c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"
Oct 30 12:57:51.517276 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.517256    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/456435f5-064d-48e8-8b14-855cf23581c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx\" (UID: \"456435f5-064d-48e8-8b14-855cf23581c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"
Oct 30 12:57:51.519438 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.519416    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/456435f5-064d-48e8-8b14-855cf23581c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx\" (UID: \"456435f5-064d-48e8-8b14-855cf23581c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"
Oct 30 12:57:51.536103 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.536080    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4p4k\" (UniqueName: \"kubernetes.io/projected/456435f5-064d-48e8-8b14-855cf23581c7-kube-api-access-v4p4k\") pod \"maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx\" (UID: \"456435f5-064d-48e8-8b14-855cf23581c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"
Oct 30 12:57:51.552573 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.552555    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"
Oct 30 12:57:51.618365 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.618293    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2983f91d-27e6-4289-80c6-0a5bb4162128-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2983f91d-27e6-4289-80c6-0a5bb4162128-maas-api-openshift-ca\") pod \"2983f91d-27e6-4289-80c6-0a5bb4162128\" (UID: \"2983f91d-27e6-4289-80c6-0a5bb4162128\") "
Oct 30 12:57:51.618560 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.618470    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2983f91d-27e6-4289-80c6-0a5bb4162128-kube-api-access-r5tcj\" (UniqueName: \"kubernetes.io/projected/2983f91d-27e6-4289-80c6-0a5bb4162128-kube-api-access-r5tcj\") pod \"2983f91d-27e6-4289-80c6-0a5bb4162128\" (UID: \"2983f91d-27e6-4289-80c6-0a5bb4162128\") "
Oct 30 12:57:51.618560 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.618528    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2983f91d-27e6-4289-80c6-0a5bb4162128-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2983f91d-27e6-4289-80c6-0a5bb4162128-maas-api-xks-ca\") pod \"2983f91d-27e6-4289-80c6-0a5bb4162128\" (UID: \"2983f91d-27e6-4289-80c6-0a5bb4162128\") "
Oct 30 12:57:51.618690 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.618646    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2983f91d-27e6-4289-80c6-0a5bb4162128-maas-api-openshift-ca" pod "2983f91d-27e6-4289-80c6-0a5bb4162128" (UID: "2983f91d-27e6-4289-80c6-0a5bb4162128"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:57:51.619306 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.619282    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2983f91d-27e6-4289-80c6-0a5bb4162128-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:51.620728 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.620699    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2983f91d-27e6-4289-80c6-0a5bb4162128-maas-api-xks-ca" pod "2983f91d-27e6-4289-80c6-0a5bb4162128" (UID: "2983f91d-27e6-4289-80c6-0a5bb4162128"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:57:51.621109 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.621092    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2983f91d-27e6-4289-80c6-0a5bb4162128-kube-api-access-r5tcj" pod "2983f91d-27e6-4289-80c6-0a5bb4162128" (UID: "2983f91d-27e6-4289-80c6-0a5bb4162128"). InnerVolumeSpecName "kube-api-access-r5tcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:51.654422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.654373    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"
Oct 30 12:57:51.722535 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.720869    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5tcj\" (UniqueName: \"kubernetes.io/projected/2983f91d-27e6-4289-80c6-0a5bb4162128-kube-api-access-r5tcj\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:51.722535 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.720915    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2983f91d-27e6-4289-80c6-0a5bb4162128-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:51.952521 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:57:51.952492    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456435f5_064d_48e8_8b14_855cf23581c7.slice/crio-758bb4f79c5bbb8608a0bbe54398d02c163227b07949d68fc528a11cbcc4cc4c WatchSource:0}: Error finding container 758bb4f79c5bbb8608a0bbe54398d02c163227b07949d68fc528a11cbcc4cc4c: Status 404 returned error can't find the container with id 758bb4f79c5bbb8608a0bbe54398d02c163227b07949d68fc528a11cbcc4cc4c
Oct 30 12:57:51.952648 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:51.952597    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"]
Oct 30 12:57:52.372375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:52.372347    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"
Oct 30 12:57:52.372840 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:52.372375    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks" event={"ID":"2983f91d-27e6-4289-80c6-0a5bb4162128","Type":"ContainerDied","Data":"e18ca7f156bf7044133cc1272a72ed3996c10e6fdb6ae7b43b21be7c69af785e"}
Oct 30 12:57:52.372840 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:52.372459    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e18ca7f156bf7044133cc1272a72ed3996c10e6fdb6ae7b43b21be7c69af785e"
Oct 30 12:57:52.374276 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:52.374251    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx" event={"ID":"456435f5-064d-48e8-8b14-855cf23581c7","Type":"ContainerStarted","Data":"1682ee503a9a3203fecb921c6d69b9d075233cc3e919b7b2d9c48d55a2a9ad63"}
Oct 30 12:57:52.374378 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:52.374290    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx" event={"ID":"456435f5-064d-48e8-8b14-855cf23581c7","Type":"ContainerStarted","Data":"758bb4f79c5bbb8608a0bbe54398d02c163227b07949d68fc528a11cbcc4cc4c"}
Oct 30 12:57:52.429199 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:52.429134    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx" podStartSLOduration=1.429113658 podStartE2EDuration="1.429113658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:52.411312467 +0000 UTC m=+1260.935596816" watchObservedRunningTime="2026-10-30 12:57:52.429113658 +0000 UTC m=+1260.953397878"
Oct 30 12:57:53.383802 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:53.383774    5837 generic.go:356] "Generic (PLEG): container finished" podID="456435f5-064d-48e8-8b14-855cf23581c7" containerID="1682ee503a9a3203fecb921c6d69b9d075233cc3e919b7b2d9c48d55a2a9ad63" exitCode=0
Oct 30 12:57:53.384215 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:53.383864    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx" event={"ID":"456435f5-064d-48e8-8b14-855cf23581c7","Type":"ContainerDied","Data":"1682ee503a9a3203fecb921c6d69b9d075233cc3e919b7b2d9c48d55a2a9ad63"}
Oct 30 12:57:54.569747 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:54.569724    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"
Oct 30 12:57:54.680791 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:54.680711    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/456435f5-064d-48e8-8b14-855cf23581c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/456435f5-064d-48e8-8b14-855cf23581c7-maas-api-openshift-ca\") pod \"456435f5-064d-48e8-8b14-855cf23581c7\" (UID: \"456435f5-064d-48e8-8b14-855cf23581c7\") "
Oct 30 12:57:54.680791 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:54.680752    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/456435f5-064d-48e8-8b14-855cf23581c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/456435f5-064d-48e8-8b14-855cf23581c7-maas-api-xks-ca\") pod \"456435f5-064d-48e8-8b14-855cf23581c7\" (UID: \"456435f5-064d-48e8-8b14-855cf23581c7\") "
Oct 30 12:57:54.680791 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:54.680787    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/456435f5-064d-48e8-8b14-855cf23581c7-kube-api-access-v4p4k\" (UniqueName: \"kubernetes.io/projected/456435f5-064d-48e8-8b14-855cf23581c7-kube-api-access-v4p4k\") pod \"456435f5-064d-48e8-8b14-855cf23581c7\" (UID: \"456435f5-064d-48e8-8b14-855cf23581c7\") "
Oct 30 12:57:54.681358 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:54.681170    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456435f5-064d-48e8-8b14-855cf23581c7-maas-api-openshift-ca" pod "456435f5-064d-48e8-8b14-855cf23581c7" (UID: "456435f5-064d-48e8-8b14-855cf23581c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:57:54.681668 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:54.681580    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/456435f5-064d-48e8-8b14-855cf23581c7-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:54.683166 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:54.683140    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456435f5-064d-48e8-8b14-855cf23581c7-maas-api-xks-ca" pod "456435f5-064d-48e8-8b14-855cf23581c7" (UID: "456435f5-064d-48e8-8b14-855cf23581c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:57:54.683246 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:54.683199    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456435f5-064d-48e8-8b14-855cf23581c7-kube-api-access-v4p4k" pod "456435f5-064d-48e8-8b14-855cf23581c7" (UID: "456435f5-064d-48e8-8b14-855cf23581c7"). InnerVolumeSpecName "kube-api-access-v4p4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:54.783288 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:54.783260    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/456435f5-064d-48e8-8b14-855cf23581c7-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:54.783288 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:54.783291    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4p4k\" (UniqueName: \"kubernetes.io/projected/456435f5-064d-48e8-8b14-855cf23581c7-kube-api-access-v4p4k\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:57:55.402442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:55.402387    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"
Oct 30 12:57:55.402442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:55.402419    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx" event={"ID":"456435f5-064d-48e8-8b14-855cf23581c7","Type":"ContainerDied","Data":"758bb4f79c5bbb8608a0bbe54398d02c163227b07949d68fc528a11cbcc4cc4c"}
Oct 30 12:57:55.402632 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:55.402462    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="758bb4f79c5bbb8608a0bbe54398d02c163227b07949d68fc528a11cbcc4cc4c"
Oct 30 12:57:55.977365 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:55.977335    5837 scope.go:122] "RemoveContainer" containerID="9dd2906f0902fc6ca562e57d97838926deee5b71c8030e6d4d1d07e397fc961e"
Oct 30 12:57:57.026221 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:57.026190    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"]
Oct 30 12:57:57.037575 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:57.037550    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2db1d4-vfmgt"]
Oct 30 12:57:57.605892 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:57:57.605859    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbdd33b4-d8b8-4273-a14d-48df02f501b8" path="/var/lib/kubelet/pods/fbdd33b4-d8b8-4273-a14d-48df02f501b8/volumes"
Oct 30 12:58:01.435186 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:01.435153    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"]
Oct 30 12:58:01.439102 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:01.439078    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a4499-zgwr8"]
Oct 30 12:58:01.631106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:01.631062    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1df34c1-6cd6-437c-b22a-030354846012" path="/var/lib/kubelet/pods/a1df34c1-6cd6-437c-b22a-030354846012/volumes"
Oct 30 12:58:03.133044 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:03.132955    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"]
Oct 30 12:58:05.823835 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:05.823793    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"]
Oct 30 12:58:05.835108 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:05.835083    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-g7jhw"]
Oct 30 12:58:06.052337 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:06.052296    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"]
Oct 30 12:58:06.060695 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:06.060657    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4033bad6-4bdkf"]
Oct 30 12:58:07.610283 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:07.610243    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56ca7ea2-1892-4ebc-91cb-8e1f51a1990d" path="/var/lib/kubelet/pods/56ca7ea2-1892-4ebc-91cb-8e1f51a1990d/volumes"
Oct 30 12:58:07.611167 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:07.611145    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f2757f-0c3e-4cdf-97a8-5eb37649adc0" path="/var/lib/kubelet/pods/b1f2757f-0c3e-4cdf-97a8-5eb37649adc0/volumes"
Oct 30 12:58:08.028484 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.028445    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"]
Oct 30 12:58:08.039695 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.039669    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a32dd90d-d5jvh"]
Oct 30 12:58:08.133866 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.133828    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229" podUID="8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b" containerName="payload-processing" containerID="cri-o://40d5564404d6db60592f79c17f5a01047f97539d472f58b71a15e86ed00782c3" gracePeriod=25
Oct 30 12:58:08.468764 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.468739    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"
Oct 30 12:58:08.530102 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.529863    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"]
Oct 30 12:58:08.530285 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.530114    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6" podUID="0312c8bf-b31d-4923-b089-13a6558da515" containerName="istio-proxy" containerID="cri-o://ee8206102c7640551e5561db1048a21a34455ed02faf3b937b9cf80f381c56dd" gracePeriod=30
Oct 30 12:58:08.531144 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.531105    5837 generic.go:356] "Generic (PLEG): container finished" podID="8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b" containerID="40d5564404d6db60592f79c17f5a01047f97539d472f58b71a15e86ed00782c3" exitCode=0
Oct 30 12:58:08.531266 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.531156    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"
Oct 30 12:58:08.531266 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.531193    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229" event={"ID":"8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b","Type":"ContainerDied","Data":"40d5564404d6db60592f79c17f5a01047f97539d472f58b71a15e86ed00782c3"}
Oct 30 12:58:08.531266 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.531255    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229" event={"ID":"8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b","Type":"ContainerDied","Data":"966a7c5afc4389be380bb8ca694242ab168384e33beaaebfa1b42b10af3e935a"}
Oct 30 12:58:08.531382 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.531289    5837 scope.go:122] "RemoveContainer" containerID="40d5564404d6db60592f79c17f5a01047f97539d472f58b71a15e86ed00782c3"
Oct 30 12:58:08.545898 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.545878    5837 scope.go:122] "RemoveContainer" containerID="40d5564404d6db60592f79c17f5a01047f97539d472f58b71a15e86ed00782c3"
Oct 30 12:58:08.546222 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:58:08.546197    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d5564404d6db60592f79c17f5a01047f97539d472f58b71a15e86ed00782c3\": container with ID starting with 40d5564404d6db60592f79c17f5a01047f97539d472f58b71a15e86ed00782c3 not found: ID does not exist" containerID="40d5564404d6db60592f79c17f5a01047f97539d472f58b71a15e86ed00782c3"
Oct 30 12:58:08.546338 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.546233    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d5564404d6db60592f79c17f5a01047f97539d472f58b71a15e86ed00782c3"} err="failed to get container status \"40d5564404d6db60592f79c17f5a01047f97539d472f58b71a15e86ed00782c3\": rpc error: code = NotFound desc = could not find container \"40d5564404d6db60592f79c17f5a01047f97539d472f58b71a15e86ed00782c3\": container with ID starting with 40d5564404d6db60592f79c17f5a01047f97539d472f58b71a15e86ed00782c3 not found: ID does not exist"
Oct 30 12:58:08.559290 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.559264    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b-plugins-config-volume\") pod \"8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b\" (UID: \"8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b\") "
Oct 30 12:58:08.559610 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.559593    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b-kube-api-access-nq9vg\" (UniqueName: \"kubernetes.io/projected/8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b-kube-api-access-nq9vg\") pod \"8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b\" (UID: \"8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b\") "
Oct 30 12:58:08.559669 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.559651    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b-plugins-config-volume" pod "8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b" (UID: "8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:08.560261 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.560232    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:08.562122 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.562095    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b-kube-api-access-nq9vg" pod "8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b" (UID: "8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b"). InnerVolumeSpecName "kube-api-access-nq9vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:08.761447 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.760014    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq9vg\" (UniqueName: \"kubernetes.io/projected/8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b-kube-api-access-nq9vg\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:08.913712 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.913675    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"]
Oct 30 12:58:08.919071 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:08.919041    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-5578c9df66-kp229"]
Oct 30 12:58:09.606125 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:09.606087    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4224b1c1-3469-4374-be0c-dcfa6036aa01" path="/var/lib/kubelet/pods/4224b1c1-3469-4374-be0c-dcfa6036aa01/volumes"
Oct 30 12:58:09.606698 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:09.606676    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b" path="/var/lib/kubelet/pods/8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b/volumes"
Oct 30 12:58:12.027743 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.027710    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"]
Oct 30 12:58:12.028495 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.028477    5837 cpu_manager.go:414] "containerMap: removing container" podUID="456435f5-064d-48e8-8b14-855cf23581c7" containerName="revoke-keys"
Oct 30 12:58:12.028495 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.028496    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="456435f5-064d-48e8-8b14-855cf23581c7" containerName="revoke-keys"
Oct 30 12:58:12.028595 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.028518    5837 cpu_manager.go:414] "containerMap: removing container" podUID="2983f91d-27e6-4289-80c6-0a5bb4162128" containerName="revoke-keys"
Oct 30 12:58:12.028595 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.028524    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2983f91d-27e6-4289-80c6-0a5bb4162128" containerName="revoke-keys"
Oct 30 12:58:12.028595 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.028535    5837 cpu_manager.go:414] "containerMap: removing container" podUID="8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b" containerName="payload-processing"
Oct 30 12:58:12.028595 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.028542    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b" containerName="payload-processing"
Oct 30 12:58:12.028744 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.028640    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2983f91d-27e6-4289-80c6-0a5bb4162128" containerName="revoke-keys"
Oct 30 12:58:12.028744 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.028649    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c848bb5-bb92-4bcb-b07b-dc1ecf60a13b" containerName="payload-processing"
Oct 30 12:58:12.028744 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.028665    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="456435f5-064d-48e8-8b14-855cf23581c7" containerName="revoke-keys"
Oct 30 12:58:12.032386 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.032369    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"
Oct 30 12:58:12.050285 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.050263    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"]
Oct 30 12:58:12.079455 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.069745    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:58:12.079455 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.069841    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:58:12.079455 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.069942    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:58:12.079455 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.070022    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:58:12.136942 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.136906    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5255ee1e-35fa-45a2-bd3a-b645167e6b44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45b2b131-4m4vs\" (UID: \"5255ee1e-35fa-45a2-bd3a-b645167e6b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"
Oct 30 12:58:12.137085 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.137064    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2829j\" (UniqueName: \"kubernetes.io/projected/5255ee1e-35fa-45a2-bd3a-b645167e6b44-kube-api-access-2829j\") pod \"maas-api-revoke-sub-45b2b131-4m4vs\" (UID: \"5255ee1e-35fa-45a2-bd3a-b645167e6b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"
Oct 30 12:58:12.137201 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.137186    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5255ee1e-35fa-45a2-bd3a-b645167e6b44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45b2b131-4m4vs\" (UID: \"5255ee1e-35fa-45a2-bd3a-b645167e6b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"
Oct 30 12:58:12.238749 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.238720    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5255ee1e-35fa-45a2-bd3a-b645167e6b44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45b2b131-4m4vs\" (UID: \"5255ee1e-35fa-45a2-bd3a-b645167e6b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"
Oct 30 12:58:12.238918 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.238823    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5255ee1e-35fa-45a2-bd3a-b645167e6b44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45b2b131-4m4vs\" (UID: \"5255ee1e-35fa-45a2-bd3a-b645167e6b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"
Oct 30 12:58:12.238993 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.238974    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2829j\" (UniqueName: \"kubernetes.io/projected/5255ee1e-35fa-45a2-bd3a-b645167e6b44-kube-api-access-2829j\") pod \"maas-api-revoke-sub-45b2b131-4m4vs\" (UID: \"5255ee1e-35fa-45a2-bd3a-b645167e6b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"
Oct 30 12:58:12.239494 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.239451    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5255ee1e-35fa-45a2-bd3a-b645167e6b44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45b2b131-4m4vs\" (UID: \"5255ee1e-35fa-45a2-bd3a-b645167e6b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"
Oct 30 12:58:12.241318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.241294    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5255ee1e-35fa-45a2-bd3a-b645167e6b44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45b2b131-4m4vs\" (UID: \"5255ee1e-35fa-45a2-bd3a-b645167e6b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"
Oct 30 12:58:12.257927 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.257901    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2829j\" (UniqueName: \"kubernetes.io/projected/5255ee1e-35fa-45a2-bd3a-b645167e6b44-kube-api-access-2829j\") pod \"maas-api-revoke-sub-45b2b131-4m4vs\" (UID: \"5255ee1e-35fa-45a2-bd3a-b645167e6b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"
Oct 30 12:58:12.346953 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.346871    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"
Oct 30 12:58:12.643076 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:12.643046    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"]
Oct 30 12:58:12.643813 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:58:12.643785    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5255ee1e_35fa_45a2_bd3a_b645167e6b44.slice/crio-8a52eb2e613f3603fc2f21419392c04f35ca92f8e5df0132ed613400fc2370be WatchSource:0}: Error finding container 8a52eb2e613f3603fc2f21419392c04f35ca92f8e5df0132ed613400fc2370be: Status 404 returned error can't find the container with id 8a52eb2e613f3603fc2f21419392c04f35ca92f8e5df0132ed613400fc2370be
Oct 30 12:58:13.650957 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.650889    5837 generic.go:356] "Generic (PLEG): container finished" podID="0312c8bf-b31d-4923-b089-13a6558da515" containerID="ee8206102c7640551e5561db1048a21a34455ed02faf3b937b9cf80f381c56dd" exitCode=0
Oct 30 12:58:13.651236 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.650967    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6" event={"ID":"0312c8bf-b31d-4923-b089-13a6558da515","Type":"ContainerDied","Data":"ee8206102c7640551e5561db1048a21a34455ed02faf3b937b9cf80f381c56dd"}
Oct 30 12:58:13.652832 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.652806    5837 generic.go:356] "Generic (PLEG): container finished" podID="5255ee1e-35fa-45a2-bd3a-b645167e6b44" containerID="ee2545b750b4b722df43e3a22a2977f94e138c2faa735cba3ce16fbe09d7a5b4" exitCode=0
Oct 30 12:58:13.652945 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.652885    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs" event={"ID":"5255ee1e-35fa-45a2-bd3a-b645167e6b44","Type":"ContainerDied","Data":"ee2545b750b4b722df43e3a22a2977f94e138c2faa735cba3ce16fbe09d7a5b4"}
Oct 30 12:58:13.652945 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.652929    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs" event={"ID":"5255ee1e-35fa-45a2-bd3a-b645167e6b44","Type":"ContainerStarted","Data":"8a52eb2e613f3603fc2f21419392c04f35ca92f8e5df0132ed613400fc2370be"}
Oct 30 12:58:13.815870 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.815844    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:58:13.887853 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.887820    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0312c8bf-b31d-4923-b089-13a6558da515-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0312c8bf-b31d-4923-b089-13a6558da515-istiod-ca-cert\") pod \"0312c8bf-b31d-4923-b089-13a6558da515\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") "
Oct 30 12:58:13.888041 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.887908    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-workload-socket\") pod \"0312c8bf-b31d-4923-b089-13a6558da515\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") "
Oct 30 12:58:13.888041 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.887956    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0312c8bf-b31d-4923-b089-13a6558da515-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0312c8bf-b31d-4923-b089-13a6558da515-istio-podinfo\") pod \"0312c8bf-b31d-4923-b089-13a6558da515\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") "
Oct 30 12:58:13.888041 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.888036    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0312c8bf-b31d-4923-b089-13a6558da515-kube-api-access-b665p\" (UniqueName: \"kubernetes.io/projected/0312c8bf-b31d-4923-b089-13a6558da515-kube-api-access-b665p\") pod \"0312c8bf-b31d-4923-b089-13a6558da515\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") "
Oct 30 12:58:13.888226 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.888091    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-credential-socket\") pod \"0312c8bf-b31d-4923-b089-13a6558da515\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") "
Oct 30 12:58:13.888226 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.888165    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-istio-data\") pod \"0312c8bf-b31d-4923-b089-13a6558da515\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") "
Oct 30 12:58:13.888226 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.888190    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0312c8bf-b31d-4923-b089-13a6558da515-istio-token\" (UniqueName: \"kubernetes.io/projected/0312c8bf-b31d-4923-b089-13a6558da515-istio-token\") pod \"0312c8bf-b31d-4923-b089-13a6558da515\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") "
Oct 30 12:58:13.888384 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.888242    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-workload-socket" pod "0312c8bf-b31d-4923-b089-13a6558da515" (UID: "0312c8bf-b31d-4923-b089-13a6558da515"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:13.888384 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.888257    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-workload-certs\") pod \"0312c8bf-b31d-4923-b089-13a6558da515\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") "
Oct 30 12:58:13.888384 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.888264    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0312c8bf-b31d-4923-b089-13a6558da515-istiod-ca-cert" pod "0312c8bf-b31d-4923-b089-13a6558da515" (UID: "0312c8bf-b31d-4923-b089-13a6558da515"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:13.889164 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.888430    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-istio-envoy\") pod \"0312c8bf-b31d-4923-b089-13a6558da515\" (UID: \"0312c8bf-b31d-4923-b089-13a6558da515\") "
Oct 30 12:58:13.889164 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.888586    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-credential-socket" pod "0312c8bf-b31d-4923-b089-13a6558da515" (UID: "0312c8bf-b31d-4923-b089-13a6558da515"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:13.889164 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.888599    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-istio-data" pod "0312c8bf-b31d-4923-b089-13a6558da515" (UID: "0312c8bf-b31d-4923-b089-13a6558da515"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:13.889164 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.889092    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0312c8bf-b31d-4923-b089-13a6558da515-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.889164 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.889114    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.889164 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.889127    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.889164 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.889141    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.889164 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.889145    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-workload-certs" pod "0312c8bf-b31d-4923-b089-13a6558da515" (UID: "0312c8bf-b31d-4923-b089-13a6558da515"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:13.890978 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.890937    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0312c8bf-b31d-4923-b089-13a6558da515-istio-podinfo" pod "0312c8bf-b31d-4923-b089-13a6558da515" (UID: "0312c8bf-b31d-4923-b089-13a6558da515"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:58:13.891132 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.891101    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0312c8bf-b31d-4923-b089-13a6558da515-istio-token" pod "0312c8bf-b31d-4923-b089-13a6558da515" (UID: "0312c8bf-b31d-4923-b089-13a6558da515"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:13.891435 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.891415    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-istio-envoy" pod "0312c8bf-b31d-4923-b089-13a6558da515" (UID: "0312c8bf-b31d-4923-b089-13a6558da515"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:13.891948 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.891925    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0312c8bf-b31d-4923-b089-13a6558da515-kube-api-access-b665p" pod "0312c8bf-b31d-4923-b089-13a6558da515" (UID: "0312c8bf-b31d-4923-b089-13a6558da515"). InnerVolumeSpecName "kube-api-access-b665p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:13.990886 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.990839    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0312c8bf-b31d-4923-b089-13a6558da515-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.990886 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.990880    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.990886 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.990889    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0312c8bf-b31d-4923-b089-13a6558da515-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.990886 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.990898    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0312c8bf-b31d-4923-b089-13a6558da515-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.990886 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:13.990906    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b665p\" (UniqueName: \"kubernetes.io/projected/0312c8bf-b31d-4923-b089-13a6558da515-kube-api-access-b665p\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:14.663142 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:14.663046    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"
Oct 30 12:58:14.663142 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:14.663112    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6" event={"ID":"0312c8bf-b31d-4923-b089-13a6558da515","Type":"ContainerDied","Data":"4d3f74f32dbb15bc8848e4c50ce1f056fd86cf69ca8854318ec7a642b0f428fe"}
Oct 30 12:58:14.663690 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:14.663186    5837 scope.go:122] "RemoveContainer" containerID="ee8206102c7640551e5561db1048a21a34455ed02faf3b937b9cf80f381c56dd"
Oct 30 12:58:14.728512 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:14.728481    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"]
Oct 30 12:58:14.738449 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:14.733719    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-v4qm6"]
Oct 30 12:58:14.833689 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:14.833658    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"
Oct 30 12:58:14.902732 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:14.902704    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5255ee1e-35fa-45a2-bd3a-b645167e6b44-kube-api-access-2829j\" (UniqueName: \"kubernetes.io/projected/5255ee1e-35fa-45a2-bd3a-b645167e6b44-kube-api-access-2829j\") pod \"5255ee1e-35fa-45a2-bd3a-b645167e6b44\" (UID: \"5255ee1e-35fa-45a2-bd3a-b645167e6b44\") "
Oct 30 12:58:14.902917 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:14.902805    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5255ee1e-35fa-45a2-bd3a-b645167e6b44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5255ee1e-35fa-45a2-bd3a-b645167e6b44-maas-api-xks-ca\") pod \"5255ee1e-35fa-45a2-bd3a-b645167e6b44\" (UID: \"5255ee1e-35fa-45a2-bd3a-b645167e6b44\") "
Oct 30 12:58:14.902917 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:14.902828    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5255ee1e-35fa-45a2-bd3a-b645167e6b44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5255ee1e-35fa-45a2-bd3a-b645167e6b44-maas-api-openshift-ca\") pod \"5255ee1e-35fa-45a2-bd3a-b645167e6b44\" (UID: \"5255ee1e-35fa-45a2-bd3a-b645167e6b44\") "
Oct 30 12:58:14.903290 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:14.903264    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5255ee1e-35fa-45a2-bd3a-b645167e6b44-maas-api-openshift-ca" pod "5255ee1e-35fa-45a2-bd3a-b645167e6b44" (UID: "5255ee1e-35fa-45a2-bd3a-b645167e6b44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:14.903425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:14.903375    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5255ee1e-35fa-45a2-bd3a-b645167e6b44-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:14.905596 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:14.905572    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5255ee1e-35fa-45a2-bd3a-b645167e6b44-kube-api-access-2829j" pod "5255ee1e-35fa-45a2-bd3a-b645167e6b44" (UID: "5255ee1e-35fa-45a2-bd3a-b645167e6b44"). InnerVolumeSpecName "kube-api-access-2829j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:14.905743 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:14.905729    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5255ee1e-35fa-45a2-bd3a-b645167e6b44-maas-api-xks-ca" pod "5255ee1e-35fa-45a2-bd3a-b645167e6b44" (UID: "5255ee1e-35fa-45a2-bd3a-b645167e6b44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:15.005034 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:15.004998    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2829j\" (UniqueName: \"kubernetes.io/projected/5255ee1e-35fa-45a2-bd3a-b645167e6b44-kube-api-access-2829j\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:15.005034 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:15.005028    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5255ee1e-35fa-45a2-bd3a-b645167e6b44-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:15.097362 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:58:15.097328    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:58:15.606219 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:15.606188    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0312c8bf-b31d-4923-b089-13a6558da515" path="/var/lib/kubelet/pods/0312c8bf-b31d-4923-b089-13a6558da515/volumes"
Oct 30 12:58:15.673482 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:15.673449    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs" event={"ID":"5255ee1e-35fa-45a2-bd3a-b645167e6b44","Type":"ContainerDied","Data":"8a52eb2e613f3603fc2f21419392c04f35ca92f8e5df0132ed613400fc2370be"}
Oct 30 12:58:15.673813 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:15.673495    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a52eb2e613f3603fc2f21419392c04f35ca92f8e5df0132ed613400fc2370be"
Oct 30 12:58:15.673813 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:15.673462    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"
Oct 30 12:58:16.904422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:16.873764    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"]
Oct 30 12:58:16.904422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:16.874859    5837 cpu_manager.go:414] "containerMap: removing container" podUID="5255ee1e-35fa-45a2-bd3a-b645167e6b44" containerName="revoke-keys"
Oct 30 12:58:16.904422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:16.874882    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5255ee1e-35fa-45a2-bd3a-b645167e6b44" containerName="revoke-keys"
Oct 30 12:58:16.904422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:16.874902    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0312c8bf-b31d-4923-b089-13a6558da515" containerName="istio-proxy"
Oct 30 12:58:16.904422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:16.874909    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0312c8bf-b31d-4923-b089-13a6558da515" containerName="istio-proxy"
Oct 30 12:58:16.904422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:16.875067    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="5255ee1e-35fa-45a2-bd3a-b645167e6b44" containerName="revoke-keys"
Oct 30 12:58:16.904422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:16.875082    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0312c8bf-b31d-4923-b089-13a6558da515" containerName="istio-proxy"
Oct 30 12:58:16.916701 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:16.916669    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"]
Oct 30 12:58:16.916948 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:16.916815    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"
Oct 30 12:58:16.937560 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:16.936383    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2bfb1b\""
Oct 30 12:58:16.947940 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:16.947834    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2bfb1b-dockercfg-957bz\""
Oct 30 12:58:17.005970 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.004713    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"]
Oct 30 12:58:17.012333 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.011140    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"
Oct 30 12:58:17.131839 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.114968    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swz9m\" (UniqueName: \"kubernetes.io/projected/1ef16201-4106-444c-8a76-7fb8f437f137-kube-api-access-swz9m\") pod \"payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx\" (UID: \"1ef16201-4106-444c-8a76-7fb8f437f137\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"
Oct 30 12:58:17.132286 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.132259    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8233dac-3352-487b-97f1-9464e07860b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr\" (UID: \"b8233dac-3352-487b-97f1-9464e07860b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"
Oct 30 12:58:17.132597 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.132575    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef16201-4106-444c-8a76-7fb8f437f137-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx\" (UID: \"1ef16201-4106-444c-8a76-7fb8f437f137\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"
Oct 30 12:58:17.132796 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.132781    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n54pm\" (UniqueName: \"kubernetes.io/projected/b8233dac-3352-487b-97f1-9464e07860b7-kube-api-access-n54pm\") pod \"payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr\" (UID: \"b8233dac-3352-487b-97f1-9464e07860b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"
Oct 30 12:58:17.133043 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.122445    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"]
Oct 30 12:58:17.234097 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.234067    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef16201-4106-444c-8a76-7fb8f437f137-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx\" (UID: \"1ef16201-4106-444c-8a76-7fb8f437f137\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"
Oct 30 12:58:17.234278 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.234139    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54pm\" (UniqueName: \"kubernetes.io/projected/b8233dac-3352-487b-97f1-9464e07860b7-kube-api-access-n54pm\") pod \"payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr\" (UID: \"b8233dac-3352-487b-97f1-9464e07860b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"
Oct 30 12:58:17.234278 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.234198    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swz9m\" (UniqueName: \"kubernetes.io/projected/1ef16201-4106-444c-8a76-7fb8f437f137-kube-api-access-swz9m\") pod \"payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx\" (UID: \"1ef16201-4106-444c-8a76-7fb8f437f137\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"
Oct 30 12:58:17.234359 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.234307    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8233dac-3352-487b-97f1-9464e07860b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr\" (UID: \"b8233dac-3352-487b-97f1-9464e07860b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"
Oct 30 12:58:17.235010 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.234981    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef16201-4106-444c-8a76-7fb8f437f137-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx\" (UID: \"1ef16201-4106-444c-8a76-7fb8f437f137\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"
Oct 30 12:58:17.235110 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.235050    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8233dac-3352-487b-97f1-9464e07860b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr\" (UID: \"b8233dac-3352-487b-97f1-9464e07860b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"
Oct 30 12:58:17.254310 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.254285    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swz9m\" (UniqueName: \"kubernetes.io/projected/1ef16201-4106-444c-8a76-7fb8f437f137-kube-api-access-swz9m\") pod \"payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx\" (UID: \"1ef16201-4106-444c-8a76-7fb8f437f137\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"
Oct 30 12:58:17.255915 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.255888    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n54pm\" (UniqueName: \"kubernetes.io/projected/b8233dac-3352-487b-97f1-9464e07860b7-kube-api-access-n54pm\") pod \"payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr\" (UID: \"b8233dac-3352-487b-97f1-9464e07860b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"
Oct 30 12:58:17.320565 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.320531    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"
Oct 30 12:58:17.380991 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.380729    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"
Oct 30 12:58:17.630064 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.630032    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"]
Oct 30 12:58:17.636786 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.636700    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"
Oct 30 12:58:17.647001 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.646893    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"]
Oct 30 12:58:17.653938 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.653648    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"]
Oct 30 12:58:17.736536 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.684253    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7a04e002\""
Oct 30 12:58:17.736536 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.684613    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7a04e002-dockercfg-nkxtx\""
Oct 30 12:58:17.736536 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.688414    5837 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:58:17.776622 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.768464    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hpsq\" (UniqueName: \"kubernetes.io/projected/aaa6a65e-517c-48f6-8d57-e1bae78e1a8a-kube-api-access-5hpsq\") pod \"payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w\" (UID: \"aaa6a65e-517c-48f6-8d57-e1bae78e1a8a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"
Oct 30 12:58:17.776622 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.768646    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa6a65e-517c-48f6-8d57-e1bae78e1a8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w\" (UID: \"aaa6a65e-517c-48f6-8d57-e1bae78e1a8a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"
Oct 30 12:58:17.886165 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.879449    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa6a65e-517c-48f6-8d57-e1bae78e1a8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w\" (UID: \"aaa6a65e-517c-48f6-8d57-e1bae78e1a8a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"
Oct 30 12:58:17.901421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.890964    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hpsq\" (UniqueName: \"kubernetes.io/projected/aaa6a65e-517c-48f6-8d57-e1bae78e1a8a-kube-api-access-5hpsq\") pod \"payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w\" (UID: \"aaa6a65e-517c-48f6-8d57-e1bae78e1a8a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"
Oct 30 12:58:17.910285 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.880318    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa6a65e-517c-48f6-8d57-e1bae78e1a8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w\" (UID: \"aaa6a65e-517c-48f6-8d57-e1bae78e1a8a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"
Oct 30 12:58:17.922310 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.922273    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx" event={"ID":"1ef16201-4106-444c-8a76-7fb8f437f137","Type":"ContainerStarted","Data":"0a553a0278d2780997070886231cb5c00d1af1aed509a826ec8c986ec7c1bd0b"}
Oct 30 12:58:17.928660 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.928593    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hpsq\" (UniqueName: \"kubernetes.io/projected/aaa6a65e-517c-48f6-8d57-e1bae78e1a8a-kube-api-access-5hpsq\") pod \"payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w\" (UID: \"aaa6a65e-517c-48f6-8d57-e1bae78e1a8a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"
Oct 30 12:58:17.990573 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.990204    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"]
Oct 30 12:58:17.991916 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:17.990506    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"
Oct 30 12:58:18.400267 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:58:18.400241    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa6a65e_517c_48f6_8d57_e1bae78e1a8a.slice/crio-89ddbef310062092b6feaaadab7f293b4016b76f5091e4d2134eb8abf6bc7417 WatchSource:0}: Error finding container 89ddbef310062092b6feaaadab7f293b4016b76f5091e4d2134eb8abf6bc7417: Status 404 returned error can't find the container with id 89ddbef310062092b6feaaadab7f293b4016b76f5091e4d2134eb8abf6bc7417
Oct 30 12:58:18.401490 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:18.401468    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"]
Oct 30 12:58:18.943328 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:18.943176    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr" event={"ID":"b8233dac-3352-487b-97f1-9464e07860b7","Type":"ContainerStarted","Data":"3a2ba6f669f02e30e92dbb37a67a4232d2c2d73f447905c56dc7b28590680d2c"}
Oct 30 12:58:18.947684 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:18.947653    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w" event={"ID":"aaa6a65e-517c-48f6-8d57-e1bae78e1a8a","Type":"ContainerStarted","Data":"1319ba39b4b7c0a8d20289a067925ccb8c3a27b6071b18b4080fa1f09f215722"}
Oct 30 12:58:18.947850 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:18.947712    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w" event={"ID":"aaa6a65e-517c-48f6-8d57-e1bae78e1a8a","Type":"ContainerStarted","Data":"89ddbef310062092b6feaaadab7f293b4016b76f5091e4d2134eb8abf6bc7417"}
Oct 30 12:58:18.947968 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:18.947945    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"
Oct 30 12:58:18.986437 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:18.985556    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w" podStartSLOduration=1.9855391660000001 podStartE2EDuration="1.985539166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:18.985002291 +0000 UTC m=+1287.509286510" watchObservedRunningTime="2026-10-30 12:58:18.985539166 +0000 UTC m=+1287.509823388"
Oct 30 12:58:20.030749 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:20.030711    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"]
Oct 30 12:58:20.040969 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:20.040935    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd0e26e-9dl2b"]
Oct 30 12:58:20.969373 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:20.969329    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr" event={"ID":"b8233dac-3352-487b-97f1-9464e07860b7","Type":"ContainerStarted","Data":"0c71b5d3091f8e725f25890e0ce989e2b4c69aaee32b7d290aa71dce0c7988b9"}
Oct 30 12:58:20.969594 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:20.969536    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"
Oct 30 12:58:20.971494 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:20.971456    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx" event={"ID":"1ef16201-4106-444c-8a76-7fb8f437f137","Type":"ContainerStarted","Data":"39f5d7b16dd7b5d78f3d981fb4bdf8bf5ec8ac960d8d09bc968c556a51a0f9f8"}
Oct 30 12:58:20.971614 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:20.971534    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"
Oct 30 12:58:21.008353 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:21.008298    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr" podStartSLOduration=2.886645021 podStartE2EDuration="5.008282096s" totalImagesPullingTime="2.121637075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:21.006793681 +0000 UTC m=+1289.531077902" watchObservedRunningTime="2026-10-30 12:58:21.008282096 +0000 UTC m=+1289.532566317"
Oct 30 12:58:21.047019 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:21.046964    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx" podStartSLOduration=2.632892294 podStartE2EDuration="5.046945928s" totalImagesPullingTime="2.414053634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:21.04628628 +0000 UTC m=+1289.570570520" watchObservedRunningTime="2026-10-30 12:58:21.046945928 +0000 UTC m=+1289.571230149"
Oct 30 12:58:21.609007 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:21.608970    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a803fc4-02c2-45f6-9c33-9153bd8f7c06" path="/var/lib/kubelet/pods/3a803fc4-02c2-45f6-9c33-9153bd8f7c06/volumes"
Oct 30 12:58:23.611528 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.611492    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"]
Oct 30 12:58:23.617626 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.617606    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.639664 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.639635    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"]
Oct 30 12:58:23.645567 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.645544    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-dea6ee-openshift-default-dockercfg-cwk9z\""
Oct 30 12:58:23.649611 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.649580    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-token\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.649739 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.649698    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-envoy\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.649739 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.649731    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-data\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.649876 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.649802    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-workload-certs\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.649876 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.649847    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zslcq\" (UniqueName: \"kubernetes.io/projected/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-kube-api-access-zslcq\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.649987 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.649897    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istiod-ca-cert\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.649987 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.649937    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-workload-socket\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.650100 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.650003    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-credential-socket\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.650100 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.650070    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-podinfo\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.751149 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.751122    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-podinfo\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.751306 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.751169    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-token\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.751306 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.751213    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-envoy\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.751306 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.751228    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-data\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.751306 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.751302    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-workload-certs\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.751527 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.751353    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zslcq\" (UniqueName: \"kubernetes.io/projected/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-kube-api-access-zslcq\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.751527 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.751489    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istiod-ca-cert\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.751633 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.751555    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-workload-socket\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.751691 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.751656    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-credential-socket\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.751818 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.751795    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-workload-certs\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.751905 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.751877    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-data\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.752059 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.752033    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-credential-socket\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.752139 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.752083    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-workload-socket\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.752335 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.752315    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istiod-ca-cert\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.753673 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.753647    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-envoy\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.753874 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.753860    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-podinfo\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.770166 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.770147    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-token\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.770291 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.770273    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zslcq\" (UniqueName: \"kubernetes.io/projected/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-kube-api-access-zslcq\") pod \"e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") " pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:23.944465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:23.943734    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:24.047549 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:24.047514    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"]
Oct 30 12:58:24.055756 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:24.055703    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-287e87c2-rn5l4"]
Oct 30 12:58:24.428573 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:24.428022    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:24.428573 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:24.428110    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:24.428573 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:24.428149    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:24.447862 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:24.447809    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"]
Oct 30 12:58:25.056575 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:25.056535    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx" event={"ID":"1eb8b513-9fce-4ddd-9555-9a61cbe430dc","Type":"ContainerStarted","Data":"637c02b83950688e877e18e8245863ba25f1074f2b2c0589ec1359c9a33a4dc3"}
Oct 30 12:58:25.057064 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:25.056598    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx" event={"ID":"1eb8b513-9fce-4ddd-9555-9a61cbe430dc","Type":"ContainerStarted","Data":"e229cf43d9a5567863ce97b13ac5bd4943e804973fcfbb5176a275c093629858"}
Oct 30 12:58:25.120797 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:25.120741    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx" podStartSLOduration=2.120726896 podStartE2EDuration="2.120726896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:25.107613427 +0000 UTC m=+1293.631897645" watchObservedRunningTime="2026-10-30 12:58:25.120726896 +0000 UTC m=+1293.645011191"
Oct 30 12:58:25.605754 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:25.605723    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1755206-b5ef-4c8e-aeaa-d90b5317e36e" path="/var/lib/kubelet/pods/c1755206-b5ef-4c8e-aeaa-d90b5317e36e/volumes"
Oct 30 12:58:25.944696 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:25.944621    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:25.950381 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:25.950357    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:26.026772 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:26.026737    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"]
Oct 30 12:58:26.030678 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:26.030655    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d491f279-9crps"]
Oct 30 12:58:26.064387 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:26.064360    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:26.065305 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:26.065289    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 12:58:27.606830 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:27.606795    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d786e2d8-d256-456c-8885-56f246e71949" path="/var/lib/kubelet/pods/d786e2d8-d256-456c-8885-56f246e71949/volumes"
Oct 30 12:58:29.182607 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.182577    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"]
Oct 30 12:58:29.188554 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.188527    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"
Oct 30 12:58:29.204068 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.204041    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"]
Oct 30 12:58:29.210347 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.210320    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-dea6ee\""
Oct 30 12:58:29.251027 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.250996    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"]
Oct 30 12:58:29.255793 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.255772    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"
Oct 30 12:58:29.264942 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.264920    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9g7n\" (UniqueName: \"kubernetes.io/projected/e348cb41-090a-4d2b-9a7a-7e1f0faa15fa-kube-api-access-j9g7n\") pod \"payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9\" (UID: \"e348cb41-090a-4d2b-9a7a-7e1f0faa15fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"
Oct 30 12:58:29.265100 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.265081    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e348cb41-090a-4d2b-9a7a-7e1f0faa15fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9\" (UID: \"e348cb41-090a-4d2b-9a7a-7e1f0faa15fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"
Oct 30 12:58:29.274578 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.274552    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"]
Oct 30 12:58:29.280702 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.280665    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-dea6ee-dockercfg-jsbnl\""
Oct 30 12:58:29.366312 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.366281    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9g7n\" (UniqueName: \"kubernetes.io/projected/e348cb41-090a-4d2b-9a7a-7e1f0faa15fa-kube-api-access-j9g7n\") pod \"payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9\" (UID: \"e348cb41-090a-4d2b-9a7a-7e1f0faa15fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"
Oct 30 12:58:29.366488 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.366359    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gh2g\" (UniqueName: \"kubernetes.io/projected/777a617c-75a3-4e91-971e-639b849b0636-kube-api-access-4gh2g\") pod \"payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft\" (UID: \"777a617c-75a3-4e91-971e-639b849b0636\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"
Oct 30 12:58:29.366488 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.366454    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e348cb41-090a-4d2b-9a7a-7e1f0faa15fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9\" (UID: \"e348cb41-090a-4d2b-9a7a-7e1f0faa15fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"
Oct 30 12:58:29.366585 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.366528    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777a617c-75a3-4e91-971e-639b849b0636-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft\" (UID: \"777a617c-75a3-4e91-971e-639b849b0636\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"
Oct 30 12:58:29.367084 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.367066    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e348cb41-090a-4d2b-9a7a-7e1f0faa15fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9\" (UID: \"e348cb41-090a-4d2b-9a7a-7e1f0faa15fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"
Oct 30 12:58:29.453479 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.423152    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9g7n\" (UniqueName: \"kubernetes.io/projected/e348cb41-090a-4d2b-9a7a-7e1f0faa15fa-kube-api-access-j9g7n\") pod \"payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9\" (UID: \"e348cb41-090a-4d2b-9a7a-7e1f0faa15fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"
Oct 30 12:58:29.486420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.483557    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gh2g\" (UniqueName: \"kubernetes.io/projected/777a617c-75a3-4e91-971e-639b849b0636-kube-api-access-4gh2g\") pod \"payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft\" (UID: \"777a617c-75a3-4e91-971e-639b849b0636\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"
Oct 30 12:58:29.486420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.483717    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777a617c-75a3-4e91-971e-639b849b0636-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft\" (UID: \"777a617c-75a3-4e91-971e-639b849b0636\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"
Oct 30 12:58:29.486420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.485157    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777a617c-75a3-4e91-971e-639b849b0636-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft\" (UID: \"777a617c-75a3-4e91-971e-639b849b0636\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"
Oct 30 12:58:29.530416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.529667    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"
Oct 30 12:58:29.531797 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.531768    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gh2g\" (UniqueName: \"kubernetes.io/projected/777a617c-75a3-4e91-971e-639b849b0636-kube-api-access-4gh2g\") pod \"payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft\" (UID: \"777a617c-75a3-4e91-971e-639b849b0636\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"
Oct 30 12:58:29.573570 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.573539    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"
Oct 30 12:58:29.952484 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:58:29.952457    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode348cb41_090a_4d2b_9a7a_7e1f0faa15fa.slice/crio-5c8c12ea46e290e3c61b861fda077c264b39bc3e9353d2baf4e512b18981a72f WatchSource:0}: Error finding container 5c8c12ea46e290e3c61b861fda077c264b39bc3e9353d2baf4e512b18981a72f: Status 404 returned error can't find the container with id 5c8c12ea46e290e3c61b861fda077c264b39bc3e9353d2baf4e512b18981a72f
Oct 30 12:58:29.953085 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.953059    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"]
Oct 30 12:58:29.962108 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.962061    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"
Oct 30 12:58:29.980833 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:29.980807    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"]
Oct 30 12:58:29.980935 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:58:29.980841    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777a617c_75a3_4e91_971e_639b849b0636.slice/crio-e3acc579ff1a38e3e52864beaee7d0d4b8786cdf26ca8fe7f4d11a8fc405b1b4 WatchSource:0}: Error finding container e3acc579ff1a38e3e52864beaee7d0d4b8786cdf26ca8fe7f4d11a8fc405b1b4: Status 404 returned error can't find the container with id e3acc579ff1a38e3e52864beaee7d0d4b8786cdf26ca8fe7f4d11a8fc405b1b4
Oct 30 12:58:30.038576 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:30.037120    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"]
Oct 30 12:58:30.038576 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:30.037161    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a349c3-jcb5c"]
Oct 30 12:58:30.113624 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:30.113585    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9" event={"ID":"e348cb41-090a-4d2b-9a7a-7e1f0faa15fa","Type":"ContainerStarted","Data":"49fd0204f949d33108c84491693e67e980cdf3139849527fc1985e97dda15ed4"}
Oct 30 12:58:30.113758 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:30.113680    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9" event={"ID":"e348cb41-090a-4d2b-9a7a-7e1f0faa15fa","Type":"ContainerStarted","Data":"5c8c12ea46e290e3c61b861fda077c264b39bc3e9353d2baf4e512b18981a72f"}
Oct 30 12:58:30.114612 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:30.114592    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"
Oct 30 12:58:30.116328 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:30.116305    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft" event={"ID":"777a617c-75a3-4e91-971e-639b849b0636","Type":"ContainerStarted","Data":"e3acc579ff1a38e3e52864beaee7d0d4b8786cdf26ca8fe7f4d11a8fc405b1b4"}
Oct 30 12:58:30.116561 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:30.116543    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"
Oct 30 12:58:30.155921 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:30.155868    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9" podStartSLOduration=1.155855467 podStartE2EDuration="1.155855467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:30.152590663 +0000 UTC m=+1298.676874913" watchObservedRunningTime="2026-10-30 12:58:30.155855467 +0000 UTC m=+1298.680139687"
Oct 30 12:58:30.193987 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:30.193940    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft" podStartSLOduration=1.193926451 podStartE2EDuration="1.193926451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:30.189326799 +0000 UTC m=+1298.713611019" watchObservedRunningTime="2026-10-30 12:58:30.193926451 +0000 UTC m=+1298.718210669"
Oct 30 12:58:31.126195 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:31.126157    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft" event={"ID":"777a617c-75a3-4e91-971e-639b849b0636","Type":"ContainerStarted","Data":"6a159b8d07aef6dcc0b7f041aaa67dfe5ed68d56b71b70e7ee09b905d17b7c54"}
Oct 30 12:58:31.607489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:31.607448    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98bce3d3-8133-4ae8-a31a-e11d1a517495" path="/var/lib/kubelet/pods/98bce3d3-8133-4ae8-a31a-e11d1a517495/volumes"
Oct 30 12:58:31.995424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:31.992657    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"
Oct 30 12:58:31.995424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:31.993722    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"
Oct 30 12:58:32.412139 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:32.412021    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"]
Oct 30 12:58:32.419639 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:32.419601    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef1186-7pjtx"]
Oct 30 12:58:33.607221 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:33.607184    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="456435f5-064d-48e8-8b14-855cf23581c7" path="/var/lib/kubelet/pods/456435f5-064d-48e8-8b14-855cf23581c7/volumes"
Oct 30 12:58:37.046521 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:37.046445    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"]
Oct 30 12:58:37.092419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:37.092238    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7435d366-ghzhn"]
Oct 30 12:58:37.637520 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:37.632373    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c49477b9-a4e4-4dad-9951-6153b32e1e3c" path="/var/lib/kubelet/pods/c49477b9-a4e4-4dad-9951-6153b32e1e3c/volumes"
Oct 30 12:58:38.944441 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:38.936367    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:39.098741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:39.078525    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:39.349620 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:39.339230    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"]
Oct 30 12:58:39.403432 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:39.399772    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" podUID="467baab9-bc62-4c3f-a034-651a6491271c" containerName="istio-proxy" containerID="cri-o://ca7c09494793fb68faf1add666af94531eb1acb5d3fe26e3e24add7eb5c15f92" gracePeriod=30
Oct 30 12:58:39.525423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:39.521592    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:39.573420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:39.561674    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": context deadline exceeded"
Oct 30 12:58:39.968419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:39.942884    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:39.969348 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:39.961933    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"]
Oct 30 12:58:40.068145 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.068103    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:58:40.118422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.089551    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"
Oct 30 12:58:40.118422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.090768    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:40.118422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.092603    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:58:40.118422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.112485    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"]
Oct 30 12:58:40.118422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.112866    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:58:40.118422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.116878    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:58:40.118422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.116958    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:58:40.118422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.117024    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:58:40.157588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.135376    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4zc9\" (UniqueName: \"kubernetes.io/projected/635c4440-4922-4821-9267-e6cff12093c2-kube-api-access-f4zc9\") pod \"maas-api-revoke-keys-e2e-api-b-70621c-2msgn\" (UID: \"635c4440-4922-4821-9267-e6cff12093c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"
Oct 30 12:58:40.157588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.135470    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635c4440-4922-4821-9267-e6cff12093c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-70621c-2msgn\" (UID: \"635c4440-4922-4821-9267-e6cff12093c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"
Oct 30 12:58:40.157588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.135525    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635c4440-4922-4821-9267-e6cff12093c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-70621c-2msgn\" (UID: \"635c4440-4922-4821-9267-e6cff12093c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"
Oct 30 12:58:40.157588 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.147961    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"]
Oct 30 12:58:40.261436 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.244071    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4zc9\" (UniqueName: \"kubernetes.io/projected/635c4440-4922-4821-9267-e6cff12093c2-kube-api-access-f4zc9\") pod \"maas-api-revoke-keys-e2e-api-b-70621c-2msgn\" (UID: \"635c4440-4922-4821-9267-e6cff12093c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"
Oct 30 12:58:40.261436 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.244144    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635c4440-4922-4821-9267-e6cff12093c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-70621c-2msgn\" (UID: \"635c4440-4922-4821-9267-e6cff12093c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"
Oct 30 12:58:40.261436 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.244196    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635c4440-4922-4821-9267-e6cff12093c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-70621c-2msgn\" (UID: \"635c4440-4922-4821-9267-e6cff12093c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"
Oct 30 12:58:40.261436 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.245145    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635c4440-4922-4821-9267-e6cff12093c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-70621c-2msgn\" (UID: \"635c4440-4922-4821-9267-e6cff12093c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"
Oct 30 12:58:40.559524 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.401000    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-750ae661-p9zwl"]
Oct 30 12:58:40.559524 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.440689    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4zc9\" (UniqueName: \"kubernetes.io/projected/635c4440-4922-4821-9267-e6cff12093c2-kube-api-access-f4zc9\") pod \"maas-api-revoke-keys-e2e-api-b-70621c-2msgn\" (UID: \"635c4440-4922-4821-9267-e6cff12093c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"
Oct 30 12:58:40.559524 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.450211    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded"
Oct 30 12:58:40.559524 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.480860    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635c4440-4922-4821-9267-e6cff12093c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-70621c-2msgn\" (UID: \"635c4440-4922-4821-9267-e6cff12093c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"
Oct 30 12:58:40.559524 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.527662    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:40.559524 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.528227    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:58:40.559524 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:40.533243    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"
Oct 30 12:58:41.070425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:41.046545    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:41.094519 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:41.092820    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:58:41.604496 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:41.603979    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:41.822172 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:41.820281    5837 patch_prober.go:29] interesting pod/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:58:41.822172 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:41.820371    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" podUID="467baab9-bc62-4c3f-a034-651a6491271c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:58:42.361423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:42.350921    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:58:42.361423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:42.355121    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:58:42.689591 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:42.689487    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:43.304514 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:43.286359    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6c89d9-9c3c-4f41-9345-104dc6adbd67" path="/var/lib/kubelet/pods/6b6c89d9-9c3c-4f41-9345-104dc6adbd67/volumes"
Oct 30 12:58:43.304514 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:58:43.286858    5837 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.686s"
Oct 30 12:58:43.304514 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:43.286940    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"
Oct 30 12:58:43.304514 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:43.287021    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"
Oct 30 12:58:43.356070 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:43.356026    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:43.356608 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:43.356579    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:58:43.775486 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:43.773061    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:58:44.298924 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:44.298887    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"]
Oct 30 12:58:44.362596 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:44.362535    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:44.410153 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:44.410116    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:58:44.450593 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:44.450546    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": context deadline exceeded"
Oct 30 12:58:44.529228 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:44.529182    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"]
Oct 30 12:58:44.769422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:44.746442    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"
Oct 30 12:58:44.798420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:44.777426    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:44.842811 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:44.842469    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f0a01c\""
Oct 30 12:58:44.876427 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:44.865268    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f0a01c-dockercfg-rjpg5\""
Oct 30 12:58:44.895842 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:44.895809    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"]
Oct 30 12:58:45.063416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:45.047729    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1cf0154-dd5a-4e12-b08f-95647d9ca4a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9\" (UID: \"f1cf0154-dd5a-4e12-b08f-95647d9ca4a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"
Oct 30 12:58:45.125366 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:45.125272    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfsdb\" (UniqueName: \"kubernetes.io/projected/f1cf0154-dd5a-4e12-b08f-95647d9ca4a1-kube-api-access-bfsdb\") pod \"payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9\" (UID: \"f1cf0154-dd5a-4e12-b08f-95647d9ca4a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"
Oct 30 12:58:45.277651 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:45.270431    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfsdb\" (UniqueName: \"kubernetes.io/projected/f1cf0154-dd5a-4e12-b08f-95647d9ca4a1-kube-api-access-bfsdb\") pod \"payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9\" (UID: \"f1cf0154-dd5a-4e12-b08f-95647d9ca4a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"
Oct 30 12:58:45.277651 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:45.270751    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1cf0154-dd5a-4e12-b08f-95647d9ca4a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9\" (UID: \"f1cf0154-dd5a-4e12-b08f-95647d9ca4a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"
Oct 30 12:58:45.277651 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:45.271849    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1cf0154-dd5a-4e12-b08f-95647d9ca4a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9\" (UID: \"f1cf0154-dd5a-4e12-b08f-95647d9ca4a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"
Oct 30 12:58:45.364862 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:45.364822    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:45.440489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:45.432507    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn" event={"ID":"635c4440-4922-4821-9267-e6cff12093c2","Type":"ContainerStarted","Data":"c5e68a04dd8f72b95f606b2774c4b47dccb1fe53da069ddef9b015208663c826"}
Oct 30 12:58:45.440489 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:45.433216    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:58:45.791422 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:45.779051    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:58:46.166477 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:46.135497    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfsdb\" (UniqueName: \"kubernetes.io/projected/f1cf0154-dd5a-4e12-b08f-95647d9ca4a1-kube-api-access-bfsdb\") pod \"payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9\" (UID: \"f1cf0154-dd5a-4e12-b08f-95647d9ca4a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"
Oct 30 12:58:46.168001 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:46.167756    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"
Oct 30 12:58:46.366973 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:46.366929    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:46.570583 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:58:46.570530    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:58:46.666651 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:46.666590    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:58:46.698886 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:58:46.698839    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467baab9_bc62_4c3f_a034_651a6491271c.slice/crio-ca7c09494793fb68faf1add666af94531eb1acb5d3fe26e3e24add7eb5c15f92.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467baab9_bc62_4c3f_a034_651a6491271c.slice/crio-conmon-ca7c09494793fb68faf1add666af94531eb1acb5d3fe26e3e24add7eb5c15f92.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 12:58:46.798613 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:46.798566    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:46.798903 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:46.798865    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" podUID="48deaa90-e66d-454f-af97-1a5462424e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:46.871982 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:46.871891    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:47.159554 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:47.098060    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:47.159554 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:47.098127    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:47.384430 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:47.381170    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:58:47.837623 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:58:47.387216    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635c4440_4922_4821_9267_e6cff12093c2.slice/crio-b232c1a77a174d8fc9e20a9feef337908deaff8708d553433c019e376285e2da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635c4440_4922_4821_9267_e6cff12093c2.slice/crio-conmon-b232c1a77a174d8fc9e20a9feef337908deaff8708d553433c019e376285e2da.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 12:58:47.902035 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:47.675924    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:47.902334 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:47.902294    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:47.902513 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:47.777555    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 12:58:48.013972 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:48.013790    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:48.133854 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:48.133756    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q" podUID="c06bb10a-6170-4c87-82e8-4695cdddc789" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 12:58:48.817419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:48.800206    5837 generic.go:356] "Generic (PLEG): container finished" podID="467baab9-bc62-4c3f-a034-651a6491271c" containerID="ca7c09494793fb68faf1add666af94531eb1acb5d3fe26e3e24add7eb5c15f92" exitCode=0
Oct 30 12:58:48.817419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:48.805905    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" event={"ID":"467baab9-bc62-4c3f-a034-651a6491271c","Type":"ContainerDied","Data":"ca7c09494793fb68faf1add666af94531eb1acb5d3fe26e3e24add7eb5c15f92"}
Oct 30 12:58:48.817419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:48.806041    5837 scope.go:122] "RemoveContainer" containerID="6a55cad6a1616762cea51709c2c8eb27400cd6310ee82b191295227aa036ccea"
Oct 30 12:58:48.973324 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:48.939477    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 12:58:49.073349 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.073306    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 12:58:49.078757 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:48.976501    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:49.082582 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.074535    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:58:49.200492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.200213    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-credential-socket\") pod \"467baab9-bc62-4c3f-a034-651a6491271c\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") "
Oct 30 12:58:49.200492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.200415    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467baab9-bc62-4c3f-a034-651a6491271c-kube-api-access-dqwwg\" (UniqueName: \"kubernetes.io/projected/467baab9-bc62-4c3f-a034-651a6491271c-kube-api-access-dqwwg\") pod \"467baab9-bc62-4c3f-a034-651a6491271c\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") "
Oct 30 12:58:49.200757 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.200586    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467baab9-bc62-4c3f-a034-651a6491271c-istio-token\" (UniqueName: \"kubernetes.io/projected/467baab9-bc62-4c3f-a034-651a6491271c-istio-token\") pod \"467baab9-bc62-4c3f-a034-651a6491271c\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") "
Oct 30 12:58:49.200757 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.200687    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-istio-data\") pod \"467baab9-bc62-4c3f-a034-651a6491271c\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") "
Oct 30 12:58:49.200866 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.200812    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/467baab9-bc62-4c3f-a034-651a6491271c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467baab9-bc62-4c3f-a034-651a6491271c-istio-podinfo\") pod \"467baab9-bc62-4c3f-a034-651a6491271c\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") "
Oct 30 12:58:49.200932 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.200893    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467baab9-bc62-4c3f-a034-651a6491271c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467baab9-bc62-4c3f-a034-651a6491271c-istiod-ca-cert\") pod \"467baab9-bc62-4c3f-a034-651a6491271c\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") "
Oct 30 12:58:49.200989 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.200968    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-workload-socket\") pod \"467baab9-bc62-4c3f-a034-651a6491271c\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") "
Oct 30 12:58:49.201130 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.201066    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-workload-certs\") pod \"467baab9-bc62-4c3f-a034-651a6491271c\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") "
Oct 30 12:58:49.201207 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.201136    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-istio-envoy\") pod \"467baab9-bc62-4c3f-a034-651a6491271c\" (UID: \"467baab9-bc62-4c3f-a034-651a6491271c\") "
Oct 30 12:58:49.236496 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.226448    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-workload-socket" pod "467baab9-bc62-4c3f-a034-651a6491271c" (UID: "467baab9-bc62-4c3f-a034-651a6491271c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:49.236496 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.226497    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-credential-socket" pod "467baab9-bc62-4c3f-a034-651a6491271c" (UID: "467baab9-bc62-4c3f-a034-651a6491271c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:49.236496 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.226740    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-workload-certs" pod "467baab9-bc62-4c3f-a034-651a6491271c" (UID: "467baab9-bc62-4c3f-a034-651a6491271c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:49.238223 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.238181    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-istio-data" pod "467baab9-bc62-4c3f-a034-651a6491271c" (UID: "467baab9-bc62-4c3f-a034-651a6491271c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:49.279762 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.279720    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467baab9-bc62-4c3f-a034-651a6491271c-istiod-ca-cert" pod "467baab9-bc62-4c3f-a034-651a6491271c" (UID: "467baab9-bc62-4c3f-a034-651a6491271c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:49.424417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.381587    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-istio-envoy" pod "467baab9-bc62-4c3f-a034-651a6491271c" (UID: "467baab9-bc62-4c3f-a034-651a6491271c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:49.461226 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.461188    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/467baab9-bc62-4c3f-a034-651a6491271c-istio-podinfo" pod "467baab9-bc62-4c3f-a034-651a6491271c" (UID: "467baab9-bc62-4c3f-a034-651a6491271c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:58:49.461491 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.461244    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467baab9-bc62-4c3f-a034-651a6491271c-istio-token" pod "467baab9-bc62-4c3f-a034-651a6491271c" (UID: "467baab9-bc62-4c3f-a034-651a6491271c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:49.461778 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.461748    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467baab9-bc62-4c3f-a034-651a6491271c-kube-api-access-dqwwg" pod "467baab9-bc62-4c3f-a034-651a6491271c" (UID: "467baab9-bc62-4c3f-a034-651a6491271c"). InnerVolumeSpecName "kube-api-access-dqwwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:49.465733 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.465710    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 12:58:49.465904 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.465885    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn" event={"ID":"635c4440-4922-4821-9267-e6cff12093c2","Type":"ContainerStarted","Data":"b232c1a77a174d8fc9e20a9feef337908deaff8708d553433c019e376285e2da"}
Oct 30 12:58:49.468260 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.468234    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:49.468375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.468265    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:49.468375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.468281    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqwwg\" (UniqueName: \"kubernetes.io/projected/467baab9-bc62-4c3f-a034-651a6491271c-kube-api-access-dqwwg\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:49.468375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.468295    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467baab9-bc62-4c3f-a034-651a6491271c-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:49.468375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.468309    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:49.468375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.468321    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467baab9-bc62-4c3f-a034-651a6491271c-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:49.468375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.468333    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467baab9-bc62-4c3f-a034-651a6491271c-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:49.468375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.468346    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:49.468375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.468360    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467baab9-bc62-4c3f-a034-651a6491271c-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:49.470119 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.470048    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 12:58:49.477567 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.477263    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 12:58:49.852885 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:49.852008    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"]
Oct 30 12:58:50.334471 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.334364    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9" event={"ID":"f1cf0154-dd5a-4e12-b08f-95647d9ca4a1","Type":"ContainerStarted","Data":"a885069c4414c1032bd18343a335ca32b96c4a0606a7251b5cd909431f8d0593"}
Oct 30 12:58:50.334753 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.334714    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9" event={"ID":"f1cf0154-dd5a-4e12-b08f-95647d9ca4a1","Type":"ContainerStarted","Data":"c1b76dea5305f2f8cda5a39ba627bbcaf73e014fc748d7bb0d002bfb3c2be5ee"}
Oct 30 12:58:50.336333 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.336314    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"
Oct 30 12:58:50.341628 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.341604    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g" event={"ID":"467baab9-bc62-4c3f-a034-651a6491271c","Type":"ContainerDied","Data":"c9fbf0daa8087246cd3fac1298f97e682bd51d324475d9296fe041d7e01befb7"}
Oct 30 12:58:50.341807 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.341792    5837 scope.go:122] "RemoveContainer" containerID="ca7c09494793fb68faf1add666af94531eb1acb5d3fe26e3e24add7eb5c15f92"
Oct 30 12:58:50.342007 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.341994    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"
Oct 30 12:58:50.350678 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.350658    5837 generic.go:356] "Generic (PLEG): container finished" podID="635c4440-4922-4821-9267-e6cff12093c2" containerID="b232c1a77a174d8fc9e20a9feef337908deaff8708d553433c019e376285e2da" exitCode=0
Oct 30 12:58:50.351638 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.350839    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn" event={"ID":"635c4440-4922-4821-9267-e6cff12093c2","Type":"ContainerDied","Data":"b232c1a77a174d8fc9e20a9feef337908deaff8708d553433c019e376285e2da"}
Oct 30 12:58:50.379485 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.379383    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9" podStartSLOduration=7.379364044 podStartE2EDuration="7.379364044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:50.375809047 +0000 UTC m=+1318.900093266" watchObservedRunningTime="2026-10-30 12:58:50.379364044 +0000 UTC m=+1318.903648266"
Oct 30 12:58:50.471448 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.467477    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"]
Oct 30 12:58:50.479419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.475099    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ef1186-openshift-default-6dc695db95-4dk7g"]
Oct 30 12:58:50.947874 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.947844    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"
Oct 30 12:58:50.983106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.981971    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/635c4440-4922-4821-9267-e6cff12093c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635c4440-4922-4821-9267-e6cff12093c2-maas-api-xks-ca\") pod \"635c4440-4922-4821-9267-e6cff12093c2\" (UID: \"635c4440-4922-4821-9267-e6cff12093c2\") "
Oct 30 12:58:50.983106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.982164    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635c4440-4922-4821-9267-e6cff12093c2-kube-api-access-f4zc9\" (UniqueName: \"kubernetes.io/projected/635c4440-4922-4821-9267-e6cff12093c2-kube-api-access-f4zc9\") pod \"635c4440-4922-4821-9267-e6cff12093c2\" (UID: \"635c4440-4922-4821-9267-e6cff12093c2\") "
Oct 30 12:58:50.983106 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.982281    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/635c4440-4922-4821-9267-e6cff12093c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635c4440-4922-4821-9267-e6cff12093c2-maas-api-openshift-ca\") pod \"635c4440-4922-4821-9267-e6cff12093c2\" (UID: \"635c4440-4922-4821-9267-e6cff12093c2\") "
Oct 30 12:58:50.995423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:50.986643    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635c4440-4922-4821-9267-e6cff12093c2-maas-api-openshift-ca" pod "635c4440-4922-4821-9267-e6cff12093c2" (UID: "635c4440-4922-4821-9267-e6cff12093c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:51.048197 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:51.048141    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635c4440-4922-4821-9267-e6cff12093c2-maas-api-xks-ca" pod "635c4440-4922-4821-9267-e6cff12093c2" (UID: "635c4440-4922-4821-9267-e6cff12093c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:51.051800 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:51.051766    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635c4440-4922-4821-9267-e6cff12093c2-kube-api-access-f4zc9" pod "635c4440-4922-4821-9267-e6cff12093c2" (UID: "635c4440-4922-4821-9267-e6cff12093c2"). InnerVolumeSpecName "kube-api-access-f4zc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:51.128476 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:51.127710    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635c4440-4922-4821-9267-e6cff12093c2-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:51.128476 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:51.127753    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635c4440-4922-4821-9267-e6cff12093c2-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:51.128476 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:51.127769    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4zc9\" (UniqueName: \"kubernetes.io/projected/635c4440-4922-4821-9267-e6cff12093c2-kube-api-access-f4zc9\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:58:51.367272 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:51.367153    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"
Oct 30 12:58:51.367272 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:51.367193    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn" event={"ID":"635c4440-4922-4821-9267-e6cff12093c2","Type":"ContainerDied","Data":"c5e68a04dd8f72b95f606b2774c4b47dccb1fe53da069ddef9b015208663c826"}
Oct 30 12:58:51.367272 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:51.367242    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5e68a04dd8f72b95f606b2774c4b47dccb1fe53da069ddef9b015208663c826"
Oct 30 12:58:51.637324 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:51.637286    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467baab9-bc62-4c3f-a034-651a6491271c" path="/var/lib/kubelet/pods/467baab9-bc62-4c3f-a034-651a6491271c/volumes"
Oct 30 12:58:52.696521 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.683634    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"]
Oct 30 12:58:52.728421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.717196    5837 cpu_manager.go:414] "containerMap: removing container" podUID="635c4440-4922-4821-9267-e6cff12093c2" containerName="revoke-keys"
Oct 30 12:58:52.728421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.717226    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635c4440-4922-4821-9267-e6cff12093c2" containerName="revoke-keys"
Oct 30 12:58:52.728421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.717286    5837 cpu_manager.go:414] "containerMap: removing container" podUID="467baab9-bc62-4c3f-a034-651a6491271c" containerName="istio-proxy"
Oct 30 12:58:52.728421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.717294    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467baab9-bc62-4c3f-a034-651a6491271c" containerName="istio-proxy"
Oct 30 12:58:52.728421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.717324    5837 cpu_manager.go:414] "containerMap: removing container" podUID="467baab9-bc62-4c3f-a034-651a6491271c" containerName="istio-proxy"
Oct 30 12:58:52.728421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.717331    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467baab9-bc62-4c3f-a034-651a6491271c" containerName="istio-proxy"
Oct 30 12:58:52.728421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.717566    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="635c4440-4922-4821-9267-e6cff12093c2" containerName="revoke-keys"
Oct 30 12:58:52.728421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.717590    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="467baab9-bc62-4c3f-a034-651a6491271c" containerName="istio-proxy"
Oct 30 12:58:52.728421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.717606    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="467baab9-bc62-4c3f-a034-651a6491271c" containerName="istio-proxy"
Oct 30 12:58:52.741418 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.735756    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"]
Oct 30 12:58:52.741418 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.735879    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:52.755902 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.755878    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f1222e-openshift-default-dockercfg-4mfmf\""
Oct 30 12:58:52.895416 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.895311    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mnbs\" (UniqueName: \"kubernetes.io/projected/d3aed56e-a69c-46a1-97e6-ea012d99e593-kube-api-access-9mnbs\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:52.895637 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.895514    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-credential-socket\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:52.895711 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.895653    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-workload-certs\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:52.895711 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.895683    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-podinfo\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:52.895806 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.895727    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-workload-socket\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:52.895806 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.895773    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-data\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:52.895898 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.895842    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3aed56e-a69c-46a1-97e6-ea012d99e593-istiod-ca-cert\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:52.927417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.895959    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-envoy\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:52.927417 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:52.895990    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-token\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.005420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.004476    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-envoy\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.005420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.004528    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-token\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.005420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.004571    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mnbs\" (UniqueName: \"kubernetes.io/projected/d3aed56e-a69c-46a1-97e6-ea012d99e593-kube-api-access-9mnbs\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.005420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.004750    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-credential-socket\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.005420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.004845    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-workload-certs\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.005420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.004871    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-podinfo\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.005420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.004911    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-workload-socket\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.005420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.004966    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-data\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.005420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.005057    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3aed56e-a69c-46a1-97e6-ea012d99e593-istiod-ca-cert\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.006188 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.006007    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3aed56e-a69c-46a1-97e6-ea012d99e593-istiod-ca-cert\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.008424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.006648    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-workload-certs\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.008424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.007436    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-credential-socket\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.008424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.007706    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-workload-socket\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.008424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.008346    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-data\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.077777 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.076988    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-token\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.131014 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.122614    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mnbs\" (UniqueName: \"kubernetes.io/projected/d3aed56e-a69c-46a1-97e6-ea012d99e593-kube-api-access-9mnbs\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.136074 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.136026    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-envoy\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.139318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.139240    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-podinfo\") pod \"e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") " pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.357164 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.357046    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:53.726864 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.726636    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"]
Oct 30 12:58:53.732297 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:58:53.732202    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3aed56e_a69c_46a1_97e6_ea012d99e593.slice/crio-67c7e8fcde34a34f9e19afd22f3457e9b93b95ef072eac9942d5f6a4c8c9b654 WatchSource:0}: Error finding container 67c7e8fcde34a34f9e19afd22f3457e9b93b95ef072eac9942d5f6a4c8c9b654: Status 404 returned error can't find the container with id 67c7e8fcde34a34f9e19afd22f3457e9b93b95ef072eac9942d5f6a4c8c9b654
Oct 30 12:58:53.735885 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.735819    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:53.736013 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.735913    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:53.736013 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:53.735955    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:54.455240 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:54.455156    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" event={"ID":"d3aed56e-a69c-46a1-97e6-ea012d99e593","Type":"ContainerStarted","Data":"c8da97fb89e6d59c66ff8b4144070c3d0054c9a22fdf1c723fdf9d8c888ad50a"}
Oct 30 12:58:54.455459 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:54.455268    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" event={"ID":"d3aed56e-a69c-46a1-97e6-ea012d99e593","Type":"ContainerStarted","Data":"67c7e8fcde34a34f9e19afd22f3457e9b93b95ef072eac9942d5f6a4c8c9b654"}
Oct 30 12:58:54.521936 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:54.521876    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" podStartSLOduration=2.521860688 podStartE2EDuration="2.521860688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:54.509202319 +0000 UTC m=+1323.033486538" watchObservedRunningTime="2026-10-30 12:58:54.521860688 +0000 UTC m=+1323.046144901"
Oct 30 12:58:55.357379 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:55.357343    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:55.364280 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:55.364250    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:55.523562 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:55.523517    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:55.525985 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:55.525682    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 12:58:56.014010 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:56.013976    5837 scope.go:122] "RemoveContainer" containerID="17bd30b68ad668ef4978e1cae00b2d8644ef4bd4c85957c902531f0cbf0c6d27"
Oct 30 12:58:56.024989 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:56.024966    5837 scope.go:122] "RemoveContainer" containerID="c0f70e80ee3dd36cdd79a9e8177ceb2dfbc2e57dc8ec3db1326eb4605a968596"
Oct 30 12:58:56.034895 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:56.034872    5837 scope.go:122] "RemoveContainer" containerID="0a9525d07d347caaeef60bf1b7579dfc18f4d903a3411f638af36b0acf2add16"
Oct 30 12:58:56.044864 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:56.044846    5837 scope.go:122] "RemoveContainer" containerID="732a7f64fdd70967eb28a65ce8ef47bae1e822866f1a950e0a43fffdc5a73aeb"
Oct 30 12:58:56.057215 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:56.057191    5837 scope.go:122] "RemoveContainer" containerID="067c09ce6723d1c49208435c2139dd84b1dca1776f58977d520afe59c4b3146c"
Oct 30 12:58:56.069959 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:56.069931    5837 scope.go:122] "RemoveContainer" containerID="69669a4f501b994837ce86b08de9aaee0a5427a44476f3e419fa0e402710462a"
Oct 30 12:58:57.596521 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.596485    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"]
Oct 30 12:58:57.626539 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.626509    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"]
Oct 30 12:58:57.627641 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.627613    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"
Oct 30 12:58:57.656273 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.656245    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:58:57.656449 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.656347    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:58:57.717488 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.717462    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:58:57.717665 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.717590    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:58:57.744673 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.744645    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l4lf\" (UniqueName: \"kubernetes.io/projected/6cca97e2-cab4-49ea-a043-885ac6d1cf13-kube-api-access-8l4lf\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss\" (UID: \"6cca97e2-cab4-49ea-a043-885ac6d1cf13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"
Oct 30 12:58:57.744820 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.744774    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cca97e2-cab4-49ea-a043-885ac6d1cf13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss\" (UID: \"6cca97e2-cab4-49ea-a043-885ac6d1cf13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"
Oct 30 12:58:57.744820 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.744807    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cca97e2-cab4-49ea-a043-885ac6d1cf13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss\" (UID: \"6cca97e2-cab4-49ea-a043-885ac6d1cf13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"
Oct 30 12:58:57.846466 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.846442    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cca97e2-cab4-49ea-a043-885ac6d1cf13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss\" (UID: \"6cca97e2-cab4-49ea-a043-885ac6d1cf13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"
Oct 30 12:58:57.846658 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.846504    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cca97e2-cab4-49ea-a043-885ac6d1cf13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss\" (UID: \"6cca97e2-cab4-49ea-a043-885ac6d1cf13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"
Oct 30 12:58:57.846799 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.846780    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l4lf\" (UniqueName: \"kubernetes.io/projected/6cca97e2-cab4-49ea-a043-885ac6d1cf13-kube-api-access-8l4lf\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss\" (UID: \"6cca97e2-cab4-49ea-a043-885ac6d1cf13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"
Oct 30 12:58:57.847163 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.847136    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cca97e2-cab4-49ea-a043-885ac6d1cf13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss\" (UID: \"6cca97e2-cab4-49ea-a043-885ac6d1cf13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"
Oct 30 12:58:57.849206 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.849179    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cca97e2-cab4-49ea-a043-885ac6d1cf13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss\" (UID: \"6cca97e2-cab4-49ea-a043-885ac6d1cf13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"
Oct 30 12:58:57.865735 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.865711    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l4lf\" (UniqueName: \"kubernetes.io/projected/6cca97e2-cab4-49ea-a043-885ac6d1cf13-kube-api-access-8l4lf\") pod \"maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss\" (UID: \"6cca97e2-cab4-49ea-a043-885ac6d1cf13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"
Oct 30 12:58:57.991529 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:57.991494    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"
Oct 30 12:58:58.313338 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.313309    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"]
Oct 30 12:58:58.313490 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:58:58.313360    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cca97e2_cab4_49ea_a043_885ac6d1cf13.slice/crio-5eeb7a989fa914ca851639dba566a361ceb4f263ecad981408ad7bad8b8b8950 WatchSource:0}: Error finding container 5eeb7a989fa914ca851639dba566a361ceb4f263ecad981408ad7bad8b8b8950: Status 404 returned error can't find the container with id 5eeb7a989fa914ca851639dba566a361ceb4f263ecad981408ad7bad8b8b8950
Oct 30 12:58:58.383887 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.383856    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"]
Oct 30 12:58:58.463520 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.463486    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"]
Oct 30 12:58:58.463700 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.463619    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.547052    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss" event={"ID":"6cca97e2-cab4-49ea-a043-885ac6d1cf13","Type":"ContainerStarted","Data":"5eeb7a989fa914ca851639dba566a361ceb4f263ecad981408ad7bad8b8b8950"}
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.556092    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-workload-certs\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.556158    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6163d41-2082-46bd-802d-38271a473e7f-istio-podinfo\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.556232    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-istio-data\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.556268    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-credential-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.556291    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6163d41-2082-46bd-802d-38271a473e7f-istio-token\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.556436    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knfnm\" (UniqueName: \"kubernetes.io/projected/a6163d41-2082-46bd-802d-38271a473e7f-kube-api-access-knfnm\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.556489    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6163d41-2082-46bd-802d-38271a473e7f-istiod-ca-cert\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.556608    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-istio-envoy\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.556730    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-workload-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.658446    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6163d41-2082-46bd-802d-38271a473e7f-istiod-ca-cert\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.658523    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-istio-envoy\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.658575    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-workload-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.658639    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-workload-certs\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.658686    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6163d41-2082-46bd-802d-38271a473e7f-istio-podinfo\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.658715    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-istio-data\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.658751    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-credential-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.658780    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6163d41-2082-46bd-802d-38271a473e7f-istio-token\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.658885    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knfnm\" (UniqueName: \"kubernetes.io/projected/a6163d41-2082-46bd-802d-38271a473e7f-kube-api-access-knfnm\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.659023    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-workload-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.659042    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-workload-certs\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.659152    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-istio-data\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.659237    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6163d41-2082-46bd-802d-38271a473e7f-istiod-ca-cert\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.659272    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-credential-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.660817    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-istio-envoy\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.661248    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6163d41-2082-46bd-802d-38271a473e7f-istio-podinfo\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.677873    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6163d41-2082-46bd-802d-38271a473e7f-istio-token\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.677899    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knfnm\" (UniqueName: \"kubernetes.io/projected/a6163d41-2082-46bd-802d-38271a473e7f-kube-api-access-knfnm\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:58.808332 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:58.790074    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:59.087136 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.043234    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"]
Oct 30 12:58:59.087136 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.046969    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32501fc6-sls9l"]
Oct 30 12:58:59.126943 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.126910    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"]
Oct 30 12:58:59.127504 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:58:59.127460    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6163d41_2082_46bd_802d_38271a473e7f.slice/crio-943a04dd1d3366b7171c25ce66c8e4d41331a18bee3fbf6897b54b4c6126f169 WatchSource:0}: Error finding container 943a04dd1d3366b7171c25ce66c8e4d41331a18bee3fbf6897b54b4c6126f169: Status 404 returned error can't find the container with id 943a04dd1d3366b7171c25ce66c8e4d41331a18bee3fbf6897b54b4c6126f169
Oct 30 12:58:59.131193 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.131154    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:59.131298 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.131236    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:59.131298 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.131278    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:59.131600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.131575    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"]
Oct 30 12:58:59.159492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.152152    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"]
Oct 30 12:58:59.159492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.152265    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"
Oct 30 12:58:59.164167 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.164132    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"]
Oct 30 12:58:59.170950 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.170919    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f1222e\""
Oct 30 12:58:59.171618 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.171592    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f1222e-dockercfg-fs2hp\""
Oct 30 12:58:59.224886 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.224862    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"]
Oct 30 12:58:59.225142 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.225126    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"
Oct 30 12:58:59.290943 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.290873    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nv8l\" (UniqueName: \"kubernetes.io/projected/c69d2915-b9d6-45e1-ba5e-60395ce8c199-kube-api-access-6nv8l\") pod \"payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg\" (UID: \"c69d2915-b9d6-45e1-ba5e-60395ce8c199\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"
Oct 30 12:58:59.291133 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.290980    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69d2915-b9d6-45e1-ba5e-60395ce8c199-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg\" (UID: \"c69d2915-b9d6-45e1-ba5e-60395ce8c199\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"
Oct 30 12:58:59.412515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.396620    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5vs\" (UniqueName: \"kubernetes.io/projected/d6b5d322-d077-443e-88cd-54067bfb8e15-kube-api-access-9z5vs\") pod \"payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8\" (UID: \"d6b5d322-d077-443e-88cd-54067bfb8e15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"
Oct 30 12:58:59.412515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.396743    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b5d322-d077-443e-88cd-54067bfb8e15-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8\" (UID: \"d6b5d322-d077-443e-88cd-54067bfb8e15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"
Oct 30 12:58:59.412515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.396810    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nv8l\" (UniqueName: \"kubernetes.io/projected/c69d2915-b9d6-45e1-ba5e-60395ce8c199-kube-api-access-6nv8l\") pod \"payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg\" (UID: \"c69d2915-b9d6-45e1-ba5e-60395ce8c199\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"
Oct 30 12:58:59.412515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.396899    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69d2915-b9d6-45e1-ba5e-60395ce8c199-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg\" (UID: \"c69d2915-b9d6-45e1-ba5e-60395ce8c199\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"
Oct 30 12:58:59.412515 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.399617    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69d2915-b9d6-45e1-ba5e-60395ce8c199-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg\" (UID: \"c69d2915-b9d6-45e1-ba5e-60395ce8c199\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"
Oct 30 12:58:59.501566 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.499625    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5vs\" (UniqueName: \"kubernetes.io/projected/d6b5d322-d077-443e-88cd-54067bfb8e15-kube-api-access-9z5vs\") pod \"payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8\" (UID: \"d6b5d322-d077-443e-88cd-54067bfb8e15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"
Oct 30 12:58:59.501566 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.499833    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b5d322-d077-443e-88cd-54067bfb8e15-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8\" (UID: \"d6b5d322-d077-443e-88cd-54067bfb8e15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"
Oct 30 12:58:59.501566 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.501380    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b5d322-d077-443e-88cd-54067bfb8e15-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8\" (UID: \"d6b5d322-d077-443e-88cd-54067bfb8e15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"
Oct 30 12:58:59.516909 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.516869    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nv8l\" (UniqueName: \"kubernetes.io/projected/c69d2915-b9d6-45e1-ba5e-60395ce8c199-kube-api-access-6nv8l\") pod \"payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg\" (UID: \"c69d2915-b9d6-45e1-ba5e-60395ce8c199\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"
Oct 30 12:58:59.522608 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.522586    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"
Oct 30 12:58:59.525733 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.525686    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5vs\" (UniqueName: \"kubernetes.io/projected/d6b5d322-d077-443e-88cd-54067bfb8e15-kube-api-access-9z5vs\") pod \"payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8\" (UID: \"d6b5d322-d077-443e-88cd-54067bfb8e15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"
Oct 30 12:58:59.564923 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.564843    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr" event={"ID":"a6163d41-2082-46bd-802d-38271a473e7f","Type":"ContainerStarted","Data":"67800d8309f8c4a7bb5120ebe83730e14ac2aae038bee7c02ef90221643da05c"}
Oct 30 12:58:59.564923 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.564899    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr" event={"ID":"a6163d41-2082-46bd-802d-38271a473e7f","Type":"ContainerStarted","Data":"943a04dd1d3366b7171c25ce66c8e4d41331a18bee3fbf6897b54b4c6126f169"}
Oct 30 12:58:59.568457 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.567321    5837 generic.go:356] "Generic (PLEG): container finished" podID="1d739dd4-b062-4a32-a496-637ee6476b02" containerID="4a055c20d83537e538b19f936d38c372844259b21dc554847af0f977d2e61d23" exitCode=0
Oct 30 12:58:59.568457 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.567414    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm" event={"ID":"1d739dd4-b062-4a32-a496-637ee6476b02","Type":"ContainerDied","Data":"4a055c20d83537e538b19f936d38c372844259b21dc554847af0f977d2e61d23"}
Oct 30 12:58:59.568457 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.567877    5837 scope.go:122] "RemoveContainer" containerID="4a055c20d83537e538b19f936d38c372844259b21dc554847af0f977d2e61d23"
Oct 30 12:58:59.571965 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.570013    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:59.571965 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.570119    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:59.571965 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.570196    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:59.584123 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.578462    5837 generic.go:356] "Generic (PLEG): container finished" podID="6cca97e2-cab4-49ea-a043-885ac6d1cf13" containerID="8548f354fb10acbfb237eb405bce8657a71c797d9ea31aa6e72e368dac6ed14f" exitCode=0
Oct 30 12:58:59.584123 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.581363    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss" event={"ID":"6cca97e2-cab4-49ea-a043-885ac6d1cf13","Type":"ContainerDied","Data":"8548f354fb10acbfb237eb405bce8657a71c797d9ea31aa6e72e368dac6ed14f"}
Oct 30 12:58:59.636095 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.636062    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c78a3d85-7386-47c5-9eb1-c7faea052546" path="/var/lib/kubelet/pods/c78a3d85-7386-47c5-9eb1-c7faea052546/volumes"
Oct 30 12:58:59.643954 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.643692    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr" podStartSLOduration=1.6436706810000001 podStartE2EDuration="1.643670681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:59.619851695 +0000 UTC m=+1328.144135915" watchObservedRunningTime="2026-10-30 12:58:59.643670681 +0000 UTC m=+1328.167954903"
Oct 30 12:58:59.790538 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.790507    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:58:59.795159 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.794936    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"
Oct 30 12:58:59.901276 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:58:59.901249    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"]
Oct 30 12:59:00.107130 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.107101    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"]
Oct 30 12:59:00.107488 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:59:00.107458    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b5d322_d077_443e_88cd_54067bfb8e15.slice/crio-46188f640906e323c8b225acda978da69b301533552f74075dfa66788f184ff5 WatchSource:0}: Error finding container 46188f640906e323c8b225acda978da69b301533552f74075dfa66788f184ff5: Status 404 returned error can't find the container with id 46188f640906e323c8b225acda978da69b301533552f74075dfa66788f184ff5
Oct 30 12:59:00.260923 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.260893    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"]
Oct 30 12:59:00.290741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.290701    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"]
Oct 30 12:59:00.680889 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.680851    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg" event={"ID":"c69d2915-b9d6-45e1-ba5e-60395ce8c199","Type":"ContainerStarted","Data":"0df2dcac8d30edc6212e4912a70c515e7405f98c5d64f935ab1b84271d5b88a9"}
Oct 30 12:59:00.681099 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.680915    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg" event={"ID":"c69d2915-b9d6-45e1-ba5e-60395ce8c199","Type":"ContainerStarted","Data":"97b4e82e0259b9ef2904e564621e5d4f8d2b78eee7e5df67f032de944247f3f6"}
Oct 30 12:59:00.681099 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.680942    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"
Oct 30 12:59:00.683084 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.683064    5837 generic.go:356] "Generic (PLEG): container finished" podID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerID="c26c377726944f54e9e85ae5869902e31227aef5e09b97fbe55dddb4cb24c7b4" exitCode=0
Oct 30 12:59:00.683151 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.683130    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4" event={"ID":"e7c0f7fa-4a94-41ae-a846-8309e089d18e","Type":"ContainerDied","Data":"c26c377726944f54e9e85ae5869902e31227aef5e09b97fbe55dddb4cb24c7b4"}
Oct 30 12:59:00.703420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.683596    5837 scope.go:122] "RemoveContainer" containerID="c26c377726944f54e9e85ae5869902e31227aef5e09b97fbe55dddb4cb24c7b4"
Oct 30 12:59:00.739419 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.717971    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:00.740554 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.740511    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:00.740794 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.740772    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:00.753719 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.753675    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg" podStartSLOduration=1.753656596 podStartE2EDuration="1.753656596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:00.745948961 +0000 UTC m=+1329.270233181" watchObservedRunningTime="2026-10-30 12:59:00.753656596 +0000 UTC m=+1329.277940817"
Oct 30 12:59:00.756923 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.756893    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm" event={"ID":"1d739dd4-b062-4a32-a496-637ee6476b02","Type":"ContainerStarted","Data":"971a01975f4a5a50e0206c16b3bfec554fa872d6363105324136b718fa902b10"}
Oct 30 12:59:00.761939 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.761915    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8" event={"ID":"d6b5d322-d077-443e-88cd-54067bfb8e15","Type":"ContainerStarted","Data":"ca8e91cc987f620aab5d0f846dc84f5d5f85093677e12fa02b9d5e576e32cc4f"}
Oct 30 12:59:00.762121 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.762106    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8" event={"ID":"d6b5d322-d077-443e-88cd-54067bfb8e15","Type":"ContainerStarted","Data":"46188f640906e323c8b225acda978da69b301533552f74075dfa66788f184ff5"}
Oct 30 12:59:00.762317 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.762297    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"
Oct 30 12:59:00.831924 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.805217    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 12:59:00.831924 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.805278    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr" podUID="a6163d41-2082-46bd-802d-38271a473e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 12:59:00.933205 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:00.933088    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8" podStartSLOduration=1.933068413 podStartE2EDuration="1.933068413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:00.906961261 +0000 UTC m=+1329.431245504" watchObservedRunningTime="2026-10-30 12:59:00.933068413 +0000 UTC m=+1329.457352635"
Oct 30 12:59:01.327073 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.327039    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:59:01.327612 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.327574    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 12:59:01.327867 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.327823    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm" podUID="1d739dd4-b062-4a32-a496-637ee6476b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 12:59:01.470510 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.470483    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"
Oct 30 12:59:01.544423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.544113    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cca97e2-cab4-49ea-a043-885ac6d1cf13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cca97e2-cab4-49ea-a043-885ac6d1cf13-maas-api-xks-ca\") pod \"6cca97e2-cab4-49ea-a043-885ac6d1cf13\" (UID: \"6cca97e2-cab4-49ea-a043-885ac6d1cf13\") "
Oct 30 12:59:01.544423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.544275    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cca97e2-cab4-49ea-a043-885ac6d1cf13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cca97e2-cab4-49ea-a043-885ac6d1cf13-maas-api-openshift-ca\") pod \"6cca97e2-cab4-49ea-a043-885ac6d1cf13\" (UID: \"6cca97e2-cab4-49ea-a043-885ac6d1cf13\") "
Oct 30 12:59:01.544423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.544336    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cca97e2-cab4-49ea-a043-885ac6d1cf13-kube-api-access-8l4lf\" (UniqueName: \"kubernetes.io/projected/6cca97e2-cab4-49ea-a043-885ac6d1cf13-kube-api-access-8l4lf\") pod \"6cca97e2-cab4-49ea-a043-885ac6d1cf13\" (UID: \"6cca97e2-cab4-49ea-a043-885ac6d1cf13\") "
Oct 30 12:59:01.547343 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.547304    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cca97e2-cab4-49ea-a043-885ac6d1cf13-maas-api-openshift-ca" pod "6cca97e2-cab4-49ea-a043-885ac6d1cf13" (UID: "6cca97e2-cab4-49ea-a043-885ac6d1cf13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:01.549354 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.549323    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cca97e2-cab4-49ea-a043-885ac6d1cf13-kube-api-access-8l4lf" pod "6cca97e2-cab4-49ea-a043-885ac6d1cf13" (UID: "6cca97e2-cab4-49ea-a043-885ac6d1cf13"). InnerVolumeSpecName "kube-api-access-8l4lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:01.553736 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.553706    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cca97e2-cab4-49ea-a043-885ac6d1cf13-maas-api-xks-ca" pod "6cca97e2-cab4-49ea-a043-885ac6d1cf13" (UID: "6cca97e2-cab4-49ea-a043-885ac6d1cf13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:59:01.649940 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.649834    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cca97e2-cab4-49ea-a043-885ac6d1cf13-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:01.649940 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.649877    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cca97e2-cab4-49ea-a043-885ac6d1cf13-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:01.649940 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.649894    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l4lf\" (UniqueName: \"kubernetes.io/projected/6cca97e2-cab4-49ea-a043-885ac6d1cf13-kube-api-access-8l4lf\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:01.833420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.829821    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 12:59:01.833420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.829881    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr" podUID="a6163d41-2082-46bd-802d-38271a473e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 12:59:01.874291 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.874255    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss" event={"ID":"6cca97e2-cab4-49ea-a043-885ac6d1cf13","Type":"ContainerDied","Data":"5eeb7a989fa914ca851639dba566a361ceb4f263ecad981408ad7bad8b8b8950"}
Oct 30 12:59:01.874573 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.874557    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eeb7a989fa914ca851639dba566a361ceb4f263ecad981408ad7bad8b8b8950"
Oct 30 12:59:01.875181 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.875160    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"
Oct 30 12:59:01.878648 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:01.878610    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4" event={"ID":"e7c0f7fa-4a94-41ae-a846-8309e089d18e","Type":"ContainerStarted","Data":"0e80c6fd2e1837884c48db5d58574f1e7254b575d10ce32e2047f0a6318d36f2"}
Oct 30 12:59:02.320426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:02.316825    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 12:59:02.320426 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:02.316888    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm" podUID="1d739dd4-b062-4a32-a496-637ee6476b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 12:59:02.459851 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:02.459426    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"
Oct 30 12:59:02.844424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:02.825305    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:59:02.844424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:02.825464    5837 patch_prober.go:29] interesting pod/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 30 12:59:02.844424 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:02.825511    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4" podUID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 30 12:59:02.855263 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:02.851755    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 12:59:02.855263 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:02.851819    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr" podUID="a6163d41-2082-46bd-802d-38271a473e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 12:59:03.000328 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:03.000276    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:59:03.319842 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:03.319805    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 12:59:03.320099 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:03.320067    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm" podUID="1d739dd4-b062-4a32-a496-637ee6476b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 12:59:03.863420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:03.828495    5837 patch_prober.go:29] interesting pod/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 30 12:59:03.863420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:03.828557    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4" podUID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 30 12:59:03.863420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:03.830909    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 12:59:03.863420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:03.830961    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr" podUID="a6163d41-2082-46bd-802d-38271a473e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 12:59:04.319892 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:04.319853    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 12:59:04.321709 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:04.321675    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm" podUID="1d739dd4-b062-4a32-a496-637ee6476b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 12:59:04.799491 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:04.790625    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 12:59:04.799491 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:04.790686    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr" podUID="a6163d41-2082-46bd-802d-38271a473e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 12:59:04.830850 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:04.830814    5837 patch_prober.go:29] interesting pod/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 30 12:59:04.831129 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:04.831096    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4" podUID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 30 12:59:05.284449 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:05.275222    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt" podUID="e9609568-7103-4f76-bbdc-877621e7487e" containerName="payload-processing" containerID="cri-o://18c9907ab7d5b68fc32173659abb34113688fa6bde329942bbe7fc1b799e6664" gracePeriod=25
Oct 30 12:59:05.300421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:05.298526    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5" podUID="d9e37dc1-7400-4517-9810-905a3cce65e2" containerName="payload-pre-processing" containerID="cri-o://f61587e901a4ed5a1b163a22c49c67d05fa3878e0ae08a5849785950c1650c56" gracePeriod=25
Oct 30 12:59:05.333420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:05.317550    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 12:59:05.333420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:05.317974    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm" podUID="1d739dd4-b062-4a32-a496-637ee6476b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 12:59:05.449370 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:05.447957    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:59:05.796899 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:05.795963    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 12:59:05.796899 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:05.796021    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr" podUID="a6163d41-2082-46bd-802d-38271a473e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 12:59:05.828253 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:05.828211    5837 patch_prober.go:29] interesting pod/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 30 12:59:05.828581 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:05.828522    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4" podUID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 30 12:59:06.080732 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.080335    5837 generic.go:356] "Generic (PLEG): container finished" podID="d9e37dc1-7400-4517-9810-905a3cce65e2" containerID="f61587e901a4ed5a1b163a22c49c67d05fa3878e0ae08a5849785950c1650c56" exitCode=0
Oct 30 12:59:06.080732 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.080610    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5" event={"ID":"d9e37dc1-7400-4517-9810-905a3cce65e2","Type":"ContainerDied","Data":"f61587e901a4ed5a1b163a22c49c67d05fa3878e0ae08a5849785950c1650c56"}
Oct 30 12:59:06.095444 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.083114    5837 generic.go:356] "Generic (PLEG): container finished" podID="e9609568-7103-4f76-bbdc-877621e7487e" containerID="18c9907ab7d5b68fc32173659abb34113688fa6bde329942bbe7fc1b799e6664" exitCode=0
Oct 30 12:59:06.095444 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.083189    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt" event={"ID":"e9609568-7103-4f76-bbdc-877621e7487e","Type":"ContainerDied","Data":"18c9907ab7d5b68fc32173659abb34113688fa6bde329942bbe7fc1b799e6664"}
Oct 30 12:59:06.318917 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.318879    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 12:59:06.326555 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.326518    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm" podUID="1d739dd4-b062-4a32-a496-637ee6476b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 12:59:06.758344 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.757358    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"
Oct 30 12:59:06.762906 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.762826    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"
Oct 30 12:59:06.827434 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.825471    5837 patch_prober.go:29] interesting pod/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 30 12:59:06.827434 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.825536    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4" podUID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 30 12:59:06.827434 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.826918    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e37dc1-7400-4517-9810-905a3cce65e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e37dc1-7400-4517-9810-905a3cce65e2-plugins-config-volume\") pod \"d9e37dc1-7400-4517-9810-905a3cce65e2\" (UID: \"d9e37dc1-7400-4517-9810-905a3cce65e2\") "
Oct 30 12:59:06.827434 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.827012    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e37dc1-7400-4517-9810-905a3cce65e2-kube-api-access-7nlp9\" (UniqueName: \"kubernetes.io/projected/d9e37dc1-7400-4517-9810-905a3cce65e2-kube-api-access-7nlp9\") pod \"d9e37dc1-7400-4517-9810-905a3cce65e2\" (UID: \"d9e37dc1-7400-4517-9810-905a3cce65e2\") "
Oct 30 12:59:06.858306 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.858237    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e37dc1-7400-4517-9810-905a3cce65e2-plugins-config-volume" pod "d9e37dc1-7400-4517-9810-905a3cce65e2" (UID: "d9e37dc1-7400-4517-9810-905a3cce65e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:06.864097 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.864049    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e37dc1-7400-4517-9810-905a3cce65e2-kube-api-access-7nlp9" pod "d9e37dc1-7400-4517-9810-905a3cce65e2" (UID: "d9e37dc1-7400-4517-9810-905a3cce65e2"). InnerVolumeSpecName "kube-api-access-7nlp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:06.866463 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.864878    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 12:59:06.866463 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.864927    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr" podUID="a6163d41-2082-46bd-802d-38271a473e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 12:59:06.959992 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.956966    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9609568-7103-4f76-bbdc-877621e7487e-kube-api-access-zcghs\" (UniqueName: \"kubernetes.io/projected/e9609568-7103-4f76-bbdc-877621e7487e-kube-api-access-zcghs\") pod \"e9609568-7103-4f76-bbdc-877621e7487e\" (UID: \"e9609568-7103-4f76-bbdc-877621e7487e\") "
Oct 30 12:59:06.959992 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.957275    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9609568-7103-4f76-bbdc-877621e7487e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9609568-7103-4f76-bbdc-877621e7487e-plugins-config-volume\") pod \"e9609568-7103-4f76-bbdc-877621e7487e\" (UID: \"e9609568-7103-4f76-bbdc-877621e7487e\") "
Oct 30 12:59:06.959992 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.958122    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e37dc1-7400-4517-9810-905a3cce65e2-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:06.959992 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.958148    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nlp9\" (UniqueName: \"kubernetes.io/projected/d9e37dc1-7400-4517-9810-905a3cce65e2-kube-api-access-7nlp9\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:06.959992 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.958631    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9609568-7103-4f76-bbdc-877621e7487e-plugins-config-volume" pod "e9609568-7103-4f76-bbdc-877621e7487e" (UID: "e9609568-7103-4f76-bbdc-877621e7487e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:06.961769 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:06.961740    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9609568-7103-4f76-bbdc-877621e7487e-kube-api-access-zcghs" pod "e9609568-7103-4f76-bbdc-877621e7487e" (UID: "e9609568-7103-4f76-bbdc-877621e7487e"). InnerVolumeSpecName "kube-api-access-zcghs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:07.060070 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.060025    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9609568-7103-4f76-bbdc-877621e7487e-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:07.060070 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.060059    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcghs\" (UniqueName: \"kubernetes.io/projected/e9609568-7103-4f76-bbdc-877621e7487e-kube-api-access-zcghs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:07.130420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.130026    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt" event={"ID":"e9609568-7103-4f76-bbdc-877621e7487e","Type":"ContainerDied","Data":"68c3161627493b2a4ebaadd5cfb03194536e6a6e1cbac1717b3dc0838bbcc658"}
Oct 30 12:59:07.130420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.130099    5837 scope.go:122] "RemoveContainer" containerID="18c9907ab7d5b68fc32173659abb34113688fa6bde329942bbe7fc1b799e6664"
Oct 30 12:59:07.130420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.130256    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"
Oct 30 12:59:07.132745 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.132602    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"
Oct 30 12:59:07.132745 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.132697    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5" event={"ID":"d9e37dc1-7400-4517-9810-905a3cce65e2","Type":"ContainerDied","Data":"deb52c1913fd835213ed9121e07c99a54cc6b7cc546a7e77f6c55fdcf49bd5e0"}
Oct 30 12:59:07.154817 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.154796    5837 scope.go:122] "RemoveContainer" containerID="f61587e901a4ed5a1b163a22c49c67d05fa3878e0ae08a5849785950c1650c56"
Oct 30 12:59:07.224600 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.224500    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"]
Oct 30 12:59:07.228775 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.228746    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-70621c-76fdbc8458-mjfqt"]
Oct 30 12:59:07.248033 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.247994    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"]
Oct 30 12:59:07.251807 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.251776    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-70621c-84fb96ff87-xwxm5"]
Oct 30 12:59:07.629423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.628774    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e37dc1-7400-4517-9810-905a3cce65e2" path="/var/lib/kubelet/pods/d9e37dc1-7400-4517-9810-905a3cce65e2/volumes"
Oct 30 12:59:07.629423 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.629212    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9609568-7103-4f76-bbdc-877621e7487e" path="/var/lib/kubelet/pods/e9609568-7103-4f76-bbdc-877621e7487e/volumes"
Oct 30 12:59:07.663234 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.663201    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:59:07.664897 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.664874    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 12:59:07.815668 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.815638    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:59:07.816579 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.816558    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:59:07.817714 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.817679    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 12:59:07.823643 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.823617    5837 patch_prober.go:29] interesting pod/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 30 12:59:07.823754 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:07.823679    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4" podUID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 30 12:59:08.069681 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:08.069648    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"]
Oct 30 12:59:08.835439 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:08.834804    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:59:08.838641 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:08.838284    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:59:11.881922 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:11.881896    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"
Oct 30 12:59:11.882301 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:11.881968    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"
Oct 30 12:59:13.070301 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:13.070249    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w" podUID="aaa6a65e-517c-48f6-8d57-e1bae78e1a8a" containerName="payload-pre-processing" containerID="cri-o://1319ba39b4b7c0a8d20289a067925ccb8c3a27b6071b18b4080fa1f09f215722" gracePeriod=25
Oct 30 12:59:13.149313 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:13.149280    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"]
Oct 30 12:59:13.181955 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:13.181926    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"]
Oct 30 12:59:13.447850 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:13.447823    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"
Oct 30 12:59:13.572448 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:13.567887    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa6a65e-517c-48f6-8d57-e1bae78e1a8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa6a65e-517c-48f6-8d57-e1bae78e1a8a-plugins-config-volume\") pod \"aaa6a65e-517c-48f6-8d57-e1bae78e1a8a\" (UID: \"aaa6a65e-517c-48f6-8d57-e1bae78e1a8a\") "
Oct 30 12:59:13.572448 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:13.568099    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa6a65e-517c-48f6-8d57-e1bae78e1a8a-kube-api-access-5hpsq\" (UniqueName: \"kubernetes.io/projected/aaa6a65e-517c-48f6-8d57-e1bae78e1a8a-kube-api-access-5hpsq\") pod \"aaa6a65e-517c-48f6-8d57-e1bae78e1a8a\" (UID: \"aaa6a65e-517c-48f6-8d57-e1bae78e1a8a\") "
Oct 30 12:59:13.572448 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:13.569607    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa6a65e-517c-48f6-8d57-e1bae78e1a8a-plugins-config-volume" pod "aaa6a65e-517c-48f6-8d57-e1bae78e1a8a" (UID: "aaa6a65e-517c-48f6-8d57-e1bae78e1a8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:13.608495 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:13.608265    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa6a65e-517c-48f6-8d57-e1bae78e1a8a-kube-api-access-5hpsq" pod "aaa6a65e-517c-48f6-8d57-e1bae78e1a8a" (UID: "aaa6a65e-517c-48f6-8d57-e1bae78e1a8a"). InnerVolumeSpecName "kube-api-access-5hpsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:13.669761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:13.669731    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa6a65e-517c-48f6-8d57-e1bae78e1a8a-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:13.669761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:13.669759    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hpsq\" (UniqueName: \"kubernetes.io/projected/aaa6a65e-517c-48f6-8d57-e1bae78e1a8a-kube-api-access-5hpsq\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:14.198654 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:14.198623    5837 generic.go:356] "Generic (PLEG): container finished" podID="aaa6a65e-517c-48f6-8d57-e1bae78e1a8a" containerID="1319ba39b4b7c0a8d20289a067925ccb8c3a27b6071b18b4080fa1f09f215722" exitCode=0
Oct 30 12:59:14.199043 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:14.198665    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"
Oct 30 12:59:14.199043 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:14.198704    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w" event={"ID":"aaa6a65e-517c-48f6-8d57-e1bae78e1a8a","Type":"ContainerDied","Data":"1319ba39b4b7c0a8d20289a067925ccb8c3a27b6071b18b4080fa1f09f215722"}
Oct 30 12:59:14.199043 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:14.198749    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w" event={"ID":"aaa6a65e-517c-48f6-8d57-e1bae78e1a8a","Type":"ContainerDied","Data":"89ddbef310062092b6feaaadab7f293b4016b76f5091e4d2134eb8abf6bc7417"}
Oct 30 12:59:14.199043 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:14.198771    5837 scope.go:122] "RemoveContainer" containerID="1319ba39b4b7c0a8d20289a067925ccb8c3a27b6071b18b4080fa1f09f215722"
Oct 30 12:59:14.209892 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:14.209868    5837 scope.go:122] "RemoveContainer" containerID="1319ba39b4b7c0a8d20289a067925ccb8c3a27b6071b18b4080fa1f09f215722"
Oct 30 12:59:14.210208 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:59:14.210189    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1319ba39b4b7c0a8d20289a067925ccb8c3a27b6071b18b4080fa1f09f215722\": container with ID starting with 1319ba39b4b7c0a8d20289a067925ccb8c3a27b6071b18b4080fa1f09f215722 not found: ID does not exist" containerID="1319ba39b4b7c0a8d20289a067925ccb8c3a27b6071b18b4080fa1f09f215722"
Oct 30 12:59:14.210281 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:14.210212    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1319ba39b4b7c0a8d20289a067925ccb8c3a27b6071b18b4080fa1f09f215722"} err="failed to get container status \"1319ba39b4b7c0a8d20289a067925ccb8c3a27b6071b18b4080fa1f09f215722\": rpc error: code = NotFound desc = could not find container \"1319ba39b4b7c0a8d20289a067925ccb8c3a27b6071b18b4080fa1f09f215722\": container with ID starting with 1319ba39b4b7c0a8d20289a067925ccb8c3a27b6071b18b4080fa1f09f215722 not found: ID does not exist"
Oct 30 12:59:14.249046 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:14.249020    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"]
Oct 30 12:59:14.259841 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:14.259818    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-x6x5w"]
Oct 30 12:59:15.102221 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:59:15.102185    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:59:15.607185 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:15.607150    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa6a65e-517c-48f6-8d57-e1bae78e1a8a" path="/var/lib/kubelet/pods/aaa6a65e-517c-48f6-8d57-e1bae78e1a8a/volumes"
Oct 30 12:59:18.028925 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.028884    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"]
Oct 30 12:59:18.037522 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.037494    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ee7e86-zf5vh"]
Oct 30 12:59:18.150058 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.150022    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9" podUID="e348cb41-090a-4d2b-9a7a-7e1f0faa15fa" containerName="payload-processing" containerID="cri-o://49fd0204f949d33108c84491693e67e980cdf3139849527fc1985e97dda15ed4" gracePeriod=25
Oct 30 12:59:18.182698 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.182663    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft" podUID="777a617c-75a3-4e91-971e-639b849b0636" containerName="payload-pre-processing" containerID="cri-o://6a159b8d07aef6dcc0b7f041aaa67dfe5ed68d56b71b70e7ee09b905d17b7c54" gracePeriod=25
Oct 30 12:59:18.269222 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.269195    5837 generic.go:356] "Generic (PLEG): container finished" podID="e348cb41-090a-4d2b-9a7a-7e1f0faa15fa" containerID="49fd0204f949d33108c84491693e67e980cdf3139849527fc1985e97dda15ed4" exitCode=0
Oct 30 12:59:18.269365 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.269266    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9" event={"ID":"e348cb41-090a-4d2b-9a7a-7e1f0faa15fa","Type":"ContainerDied","Data":"49fd0204f949d33108c84491693e67e980cdf3139849527fc1985e97dda15ed4"}
Oct 30 12:59:18.557644 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.557619    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"
Oct 30 12:59:18.601235 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.601207    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"
Oct 30 12:59:18.648859 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.648824    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777a617c-75a3-4e91-971e-639b849b0636-kube-api-access-4gh2g\" (UniqueName: \"kubernetes.io/projected/777a617c-75a3-4e91-971e-639b849b0636-kube-api-access-4gh2g\") pod \"777a617c-75a3-4e91-971e-639b849b0636\" (UID: \"777a617c-75a3-4e91-971e-639b849b0636\") "
Oct 30 12:59:18.649062 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.649043    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e348cb41-090a-4d2b-9a7a-7e1f0faa15fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e348cb41-090a-4d2b-9a7a-7e1f0faa15fa-plugins-config-volume\") pod \"e348cb41-090a-4d2b-9a7a-7e1f0faa15fa\" (UID: \"e348cb41-090a-4d2b-9a7a-7e1f0faa15fa\") "
Oct 30 12:59:18.649144 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.649115    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e348cb41-090a-4d2b-9a7a-7e1f0faa15fa-kube-api-access-j9g7n\" (UniqueName: \"kubernetes.io/projected/e348cb41-090a-4d2b-9a7a-7e1f0faa15fa-kube-api-access-j9g7n\") pod \"e348cb41-090a-4d2b-9a7a-7e1f0faa15fa\" (UID: \"e348cb41-090a-4d2b-9a7a-7e1f0faa15fa\") "
Oct 30 12:59:18.649329 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.649311    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777a617c-75a3-4e91-971e-639b849b0636-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777a617c-75a3-4e91-971e-639b849b0636-plugins-config-volume\") pod \"777a617c-75a3-4e91-971e-639b849b0636\" (UID: \"777a617c-75a3-4e91-971e-639b849b0636\") "
Oct 30 12:59:18.650374 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.650345    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777a617c-75a3-4e91-971e-639b849b0636-plugins-config-volume" pod "777a617c-75a3-4e91-971e-639b849b0636" (UID: "777a617c-75a3-4e91-971e-639b849b0636"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:18.651167 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.651138    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e348cb41-090a-4d2b-9a7a-7e1f0faa15fa-plugins-config-volume" pod "e348cb41-090a-4d2b-9a7a-7e1f0faa15fa" (UID: "e348cb41-090a-4d2b-9a7a-7e1f0faa15fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:18.653148 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.653113    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777a617c-75a3-4e91-971e-639b849b0636-kube-api-access-4gh2g" pod "777a617c-75a3-4e91-971e-639b849b0636" (UID: "777a617c-75a3-4e91-971e-639b849b0636"). InnerVolumeSpecName "kube-api-access-4gh2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:18.653373 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.653348    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e348cb41-090a-4d2b-9a7a-7e1f0faa15fa-kube-api-access-j9g7n" pod "e348cb41-090a-4d2b-9a7a-7e1f0faa15fa" (UID: "e348cb41-090a-4d2b-9a7a-7e1f0faa15fa"). InnerVolumeSpecName "kube-api-access-j9g7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:18.751928 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.751898    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gh2g\" (UniqueName: \"kubernetes.io/projected/777a617c-75a3-4e91-971e-639b849b0636-kube-api-access-4gh2g\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:18.751928 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.751928    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e348cb41-090a-4d2b-9a7a-7e1f0faa15fa-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:18.752122 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.751937    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9g7n\" (UniqueName: \"kubernetes.io/projected/e348cb41-090a-4d2b-9a7a-7e1f0faa15fa-kube-api-access-j9g7n\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:18.752122 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:18.751946    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777a617c-75a3-4e91-971e-639b849b0636-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:19.278667 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.278634    5837 generic.go:356] "Generic (PLEG): container finished" podID="777a617c-75a3-4e91-971e-639b849b0636" containerID="6a159b8d07aef6dcc0b7f041aaa67dfe5ed68d56b71b70e7ee09b905d17b7c54" exitCode=0
Oct 30 12:59:19.279071 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.278684    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"
Oct 30 12:59:19.279071 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.278721    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft" event={"ID":"777a617c-75a3-4e91-971e-639b849b0636","Type":"ContainerDied","Data":"6a159b8d07aef6dcc0b7f041aaa67dfe5ed68d56b71b70e7ee09b905d17b7c54"}
Oct 30 12:59:19.279071 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.278777    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft" event={"ID":"777a617c-75a3-4e91-971e-639b849b0636","Type":"ContainerDied","Data":"e3acc579ff1a38e3e52864beaee7d0d4b8786cdf26ca8fe7f4d11a8fc405b1b4"}
Oct 30 12:59:19.279071 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.278798    5837 scope.go:122] "RemoveContainer" containerID="6a159b8d07aef6dcc0b7f041aaa67dfe5ed68d56b71b70e7ee09b905d17b7c54"
Oct 30 12:59:19.280475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.280451    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9" event={"ID":"e348cb41-090a-4d2b-9a7a-7e1f0faa15fa","Type":"ContainerDied","Data":"5c8c12ea46e290e3c61b861fda077c264b39bc3e9353d2baf4e512b18981a72f"}
Oct 30 12:59:19.280475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.280475    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"
Oct 30 12:59:19.296411 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.292664    5837 scope.go:122] "RemoveContainer" containerID="6a159b8d07aef6dcc0b7f041aaa67dfe5ed68d56b71b70e7ee09b905d17b7c54"
Oct 30 12:59:19.296968 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:59:19.296935    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a159b8d07aef6dcc0b7f041aaa67dfe5ed68d56b71b70e7ee09b905d17b7c54\": container with ID starting with 6a159b8d07aef6dcc0b7f041aaa67dfe5ed68d56b71b70e7ee09b905d17b7c54 not found: ID does not exist" containerID="6a159b8d07aef6dcc0b7f041aaa67dfe5ed68d56b71b70e7ee09b905d17b7c54"
Oct 30 12:59:19.297061 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.296980    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a159b8d07aef6dcc0b7f041aaa67dfe5ed68d56b71b70e7ee09b905d17b7c54"} err="failed to get container status \"6a159b8d07aef6dcc0b7f041aaa67dfe5ed68d56b71b70e7ee09b905d17b7c54\": rpc error: code = NotFound desc = could not find container \"6a159b8d07aef6dcc0b7f041aaa67dfe5ed68d56b71b70e7ee09b905d17b7c54\": container with ID starting with 6a159b8d07aef6dcc0b7f041aaa67dfe5ed68d56b71b70e7ee09b905d17b7c54 not found: ID does not exist"
Oct 30 12:59:19.297061 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.297005    5837 scope.go:122] "RemoveContainer" containerID="49fd0204f949d33108c84491693e67e980cdf3139849527fc1985e97dda15ed4"
Oct 30 12:59:19.330331 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.330305    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"]
Oct 30 12:59:19.341622 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.341600    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-6fb4b499c9-6c8ft"]
Oct 30 12:59:19.387349 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.387321    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"]
Oct 30 12:59:19.392263 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.392238    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-5bffbfd99b-w2rg9"]
Oct 30 12:59:19.607064 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.607038    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777a617c-75a3-4e91-971e-639b849b0636" path="/var/lib/kubelet/pods/777a617c-75a3-4e91-971e-639b849b0636/volumes"
Oct 30 12:59:19.607517 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.607496    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13ea519-9c63-4fb4-abc1-98f4eca8b218" path="/var/lib/kubelet/pods/c13ea519-9c63-4fb4-abc1-98f4eca8b218/volumes"
Oct 30 12:59:19.607875 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:19.607862    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e348cb41-090a-4d2b-9a7a-7e1f0faa15fa" path="/var/lib/kubelet/pods/e348cb41-090a-4d2b-9a7a-7e1f0faa15fa/volumes"
Oct 30 12:59:26.351525 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:26.351493    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"]
Oct 30 12:59:26.363507 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:26.363475    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a04e002-tn2ss"]
Oct 30 12:59:27.606869 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:27.606834    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cca97e2-cab4-49ea-a043-885ac6d1cf13" path="/var/lib/kubelet/pods/6cca97e2-cab4-49ea-a043-885ac6d1cf13/volumes"
Oct 30 12:59:28.027818 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:28.027787    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"]
Oct 30 12:59:28.038886 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:28.038864    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa897177-2j8bj"]
Oct 30 12:59:28.696539 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:28.696489    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"]
Oct 30 12:59:28.707475 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:28.707448    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-70621c-2msgn"]
Oct 30 12:59:29.606054 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:29.606025    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d339443-09e7-4990-8fcc-bfd0e6532dbe" path="/var/lib/kubelet/pods/0d339443-09e7-4990-8fcc-bfd0e6532dbe/volumes"
Oct 30 12:59:29.606533 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:29.606519    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635c4440-4922-4821-9267-e6cff12093c2" path="/var/lib/kubelet/pods/635c4440-4922-4821-9267-e6cff12093c2/volumes"
Oct 30 12:59:29.651783 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:29.651757    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"]
Oct 30 12:59:32.335741 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.318454    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"]
Oct 30 12:59:32.354761 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.354534    5837 cpu_manager.go:414] "containerMap: removing container" podUID="e348cb41-090a-4d2b-9a7a-7e1f0faa15fa" containerName="payload-processing"
Oct 30 12:59:32.354998 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.354979    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e348cb41-090a-4d2b-9a7a-7e1f0faa15fa" containerName="payload-processing"
Oct 30 12:59:32.355177 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.355164    5837 cpu_manager.go:414] "containerMap: removing container" podUID="e9609568-7103-4f76-bbdc-877621e7487e" containerName="payload-processing"
Oct 30 12:59:32.384579 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.384543    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9609568-7103-4f76-bbdc-877621e7487e" containerName="payload-processing"
Oct 30 12:59:32.384855 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.384834    5837 cpu_manager.go:414] "containerMap: removing container" podUID="aaa6a65e-517c-48f6-8d57-e1bae78e1a8a" containerName="payload-pre-processing"
Oct 30 12:59:32.384965 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.384952    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa6a65e-517c-48f6-8d57-e1bae78e1a8a" containerName="payload-pre-processing"
Oct 30 12:59:32.385111 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.385100    5837 cpu_manager.go:414] "containerMap: removing container" podUID="d9e37dc1-7400-4517-9810-905a3cce65e2" containerName="payload-pre-processing"
Oct 30 12:59:32.385198 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.385188    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e37dc1-7400-4517-9810-905a3cce65e2" containerName="payload-pre-processing"
Oct 30 12:59:32.385301 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.385291    5837 cpu_manager.go:414] "containerMap: removing container" podUID="6cca97e2-cab4-49ea-a043-885ac6d1cf13" containerName="revoke-keys"
Oct 30 12:59:32.385382 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.385372    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cca97e2-cab4-49ea-a043-885ac6d1cf13" containerName="revoke-keys"
Oct 30 12:59:32.387132 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.387096    5837 cpu_manager.go:414] "containerMap: removing container" podUID="777a617c-75a3-4e91-971e-639b849b0636" containerName="payload-pre-processing"
Oct 30 12:59:32.387290 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.387274    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777a617c-75a3-4e91-971e-639b849b0636" containerName="payload-pre-processing"
Oct 30 12:59:32.387837 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.387815    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa6a65e-517c-48f6-8d57-e1bae78e1a8a" containerName="payload-pre-processing"
Oct 30 12:59:32.387993 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.387980    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e37dc1-7400-4517-9810-905a3cce65e2" containerName="payload-pre-processing"
Oct 30 12:59:32.388089 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.388080    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cca97e2-cab4-49ea-a043-885ac6d1cf13" containerName="revoke-keys"
Oct 30 12:59:32.388185 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.388175    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="777a617c-75a3-4e91-971e-639b849b0636" containerName="payload-pre-processing"
Oct 30 12:59:32.388271 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.388262    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e348cb41-090a-4d2b-9a7a-7e1f0faa15fa" containerName="payload-processing"
Oct 30 12:59:32.393087 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.388360    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9609568-7103-4f76-bbdc-877621e7487e" containerName="payload-processing"
Oct 30 12:59:32.462189 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.462151    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"]
Oct 30 12:59:32.462457 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.462301    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"
Oct 30 12:59:32.470252 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.470148    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"]
Oct 30 12:59:32.484451 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.476491    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"
Oct 30 12:59:32.519603 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.497855    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-dea6ee\""
Oct 30 12:59:32.519603 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.498019    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-dea6ee-dockercfg-kwls5\""
Oct 30 12:59:32.519603 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.499687    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8baef900-5f0d-482f-8806-db129254c21b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b\" (UID: \"8baef900-5f0d-482f-8806-db129254c21b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"
Oct 30 12:59:32.519603 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.500004    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8djv7\" (UniqueName: \"kubernetes.io/projected/8baef900-5f0d-482f-8806-db129254c21b-kube-api-access-8djv7\") pod \"payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b\" (UID: \"8baef900-5f0d-482f-8806-db129254c21b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"
Oct 30 12:59:32.519943 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.519787    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"]
Oct 30 12:59:32.602700 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.602572    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33b2398-1559-4ffd-91a9-3d60d2a07778-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2\" (UID: \"c33b2398-1559-4ffd-91a9-3d60d2a07778\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"
Oct 30 12:59:32.603442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.603373    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8baef900-5f0d-482f-8806-db129254c21b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b\" (UID: \"8baef900-5f0d-482f-8806-db129254c21b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"
Oct 30 12:59:32.605276 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.605247    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qh4m\" (UniqueName: \"kubernetes.io/projected/c33b2398-1559-4ffd-91a9-3d60d2a07778-kube-api-access-2qh4m\") pod \"payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2\" (UID: \"c33b2398-1559-4ffd-91a9-3d60d2a07778\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"
Oct 30 12:59:32.605386 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.605347    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djv7\" (UniqueName: \"kubernetes.io/projected/8baef900-5f0d-482f-8806-db129254c21b-kube-api-access-8djv7\") pod \"payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b\" (UID: \"8baef900-5f0d-482f-8806-db129254c21b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"
Oct 30 12:59:32.607967 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.607944    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8baef900-5f0d-482f-8806-db129254c21b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b\" (UID: \"8baef900-5f0d-482f-8806-db129254c21b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"
Oct 30 12:59:32.628002 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.627957    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8djv7\" (UniqueName: \"kubernetes.io/projected/8baef900-5f0d-482f-8806-db129254c21b-kube-api-access-8djv7\") pod \"payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b\" (UID: \"8baef900-5f0d-482f-8806-db129254c21b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"
Oct 30 12:59:32.707321 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.707281    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qh4m\" (UniqueName: \"kubernetes.io/projected/c33b2398-1559-4ffd-91a9-3d60d2a07778-kube-api-access-2qh4m\") pod \"payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2\" (UID: \"c33b2398-1559-4ffd-91a9-3d60d2a07778\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"
Oct 30 12:59:32.707573 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.707557    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33b2398-1559-4ffd-91a9-3d60d2a07778-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2\" (UID: \"c33b2398-1559-4ffd-91a9-3d60d2a07778\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"
Oct 30 12:59:32.708233 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.708207    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33b2398-1559-4ffd-91a9-3d60d2a07778-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2\" (UID: \"c33b2398-1559-4ffd-91a9-3d60d2a07778\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"
Oct 30 12:59:32.727884 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.727860    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qh4m\" (UniqueName: \"kubernetes.io/projected/c33b2398-1559-4ffd-91a9-3d60d2a07778-kube-api-access-2qh4m\") pod \"payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2\" (UID: \"c33b2398-1559-4ffd-91a9-3d60d2a07778\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"
Oct 30 12:59:32.823644 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.823615    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"
Oct 30 12:59:32.827687 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:32.827662    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"
Oct 30 12:59:33.155097 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:33.155068    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"]
Oct 30 12:59:33.155226 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:59:33.155081    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8baef900_5f0d_482f_8806_db129254c21b.slice/crio-dac6988f71dac73e6e3d5e27dfb5a9eb7803ca77400b27afac0191b8f670132a WatchSource:0}: Error finding container dac6988f71dac73e6e3d5e27dfb5a9eb7803ca77400b27afac0191b8f670132a: Status 404 returned error can't find the container with id dac6988f71dac73e6e3d5e27dfb5a9eb7803ca77400b27afac0191b8f670132a
Oct 30 12:59:33.185985 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:59:33.185957    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33b2398_1559_4ffd_91a9_3d60d2a07778.slice/crio-f605751fa1132f15db4678a3a5a3f4506513be41378eb898971f50177d91be32 WatchSource:0}: Error finding container f605751fa1132f15db4678a3a5a3f4506513be41378eb898971f50177d91be32: Status 404 returned error can't find the container with id f605751fa1132f15db4678a3a5a3f4506513be41378eb898971f50177d91be32
Oct 30 12:59:33.188648 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:33.188626    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"]
Oct 30 12:59:33.472627 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:33.472524    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b" event={"ID":"8baef900-5f0d-482f-8806-db129254c21b","Type":"ContainerStarted","Data":"d1c487b8fe2fa69b1686e84af33a0047327b3c036fec9e33daf9f78b41590530"}
Oct 30 12:59:33.472627 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:33.472589    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"
Oct 30 12:59:33.472627 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:33.472615    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b" event={"ID":"8baef900-5f0d-482f-8806-db129254c21b","Type":"ContainerStarted","Data":"dac6988f71dac73e6e3d5e27dfb5a9eb7803ca77400b27afac0191b8f670132a"}
Oct 30 12:59:33.474356 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:33.474328    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2" event={"ID":"c33b2398-1559-4ffd-91a9-3d60d2a07778","Type":"ContainerStarted","Data":"c3bbaddcd5283d659060964af43b085609ec98d61ede1841d0bb8e6f16be9b59"}
Oct 30 12:59:33.474545 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:33.474371    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2" event={"ID":"c33b2398-1559-4ffd-91a9-3d60d2a07778","Type":"ContainerStarted","Data":"f605751fa1132f15db4678a3a5a3f4506513be41378eb898971f50177d91be32"}
Oct 30 12:59:33.474545 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:33.474469    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"
Oct 30 12:59:33.515281 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:33.515232    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b" podStartSLOduration=1.515218476 podStartE2EDuration="1.515218476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:33.511684159 +0000 UTC m=+1362.035968379" watchObservedRunningTime="2026-10-30 12:59:33.515218476 +0000 UTC m=+1362.039502694"
Oct 30 12:59:33.558525 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:33.558465    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2" podStartSLOduration=1.558452454 podStartE2EDuration="1.558452454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:33.555759335 +0000 UTC m=+1362.080043582" watchObservedRunningTime="2026-10-30 12:59:33.558452454 +0000 UTC m=+1362.082736673"
Oct 30 12:59:34.652630 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:34.652577    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9" podUID="f1cf0154-dd5a-4e12-b08f-95647d9ca4a1" containerName="payload-pre-processing" containerID="cri-o://a885069c4414c1032bd18343a335ca32b96c4a0606a7251b5cd909431f8d0593" gracePeriod=25
Oct 30 12:59:35.020826 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.020696    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"
Oct 30 12:59:35.150002 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.149936    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1cf0154-dd5a-4e12-b08f-95647d9ca4a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1cf0154-dd5a-4e12-b08f-95647d9ca4a1-plugins-config-volume\") pod \"f1cf0154-dd5a-4e12-b08f-95647d9ca4a1\" (UID: \"f1cf0154-dd5a-4e12-b08f-95647d9ca4a1\") "
Oct 30 12:59:35.150788 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.150194    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1cf0154-dd5a-4e12-b08f-95647d9ca4a1-kube-api-access-bfsdb\" (UniqueName: \"kubernetes.io/projected/f1cf0154-dd5a-4e12-b08f-95647d9ca4a1-kube-api-access-bfsdb\") pod \"f1cf0154-dd5a-4e12-b08f-95647d9ca4a1\" (UID: \"f1cf0154-dd5a-4e12-b08f-95647d9ca4a1\") "
Oct 30 12:59:35.151448 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.151366    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1cf0154-dd5a-4e12-b08f-95647d9ca4a1-plugins-config-volume" pod "f1cf0154-dd5a-4e12-b08f-95647d9ca4a1" (UID: "f1cf0154-dd5a-4e12-b08f-95647d9ca4a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:35.157873 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.157835    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1cf0154-dd5a-4e12-b08f-95647d9ca4a1-kube-api-access-bfsdb" pod "f1cf0154-dd5a-4e12-b08f-95647d9ca4a1" (UID: "f1cf0154-dd5a-4e12-b08f-95647d9ca4a1"). InnerVolumeSpecName "kube-api-access-bfsdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:35.252847 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.252775    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1cf0154-dd5a-4e12-b08f-95647d9ca4a1-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:35.252847 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.252800    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfsdb\" (UniqueName: \"kubernetes.io/projected/f1cf0154-dd5a-4e12-b08f-95647d9ca4a1-kube-api-access-bfsdb\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:35.327574 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.327531    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"]
Oct 30 12:59:35.327805 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.327778    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4" podUID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerName="istio-proxy" containerID="cri-o://0e80c6fd2e1837884c48db5d58574f1e7254b575d10ce32e2047f0a6318d36f2" gracePeriod=30
Oct 30 12:59:35.548492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.542550    5837 generic.go:356] "Generic (PLEG): container finished" podID="f1cf0154-dd5a-4e12-b08f-95647d9ca4a1" containerID="a885069c4414c1032bd18343a335ca32b96c4a0606a7251b5cd909431f8d0593" exitCode=0
Oct 30 12:59:35.548492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.542766    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9" event={"ID":"f1cf0154-dd5a-4e12-b08f-95647d9ca4a1","Type":"ContainerDied","Data":"a885069c4414c1032bd18343a335ca32b96c4a0606a7251b5cd909431f8d0593"}
Oct 30 12:59:35.548492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.542829    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9" event={"ID":"f1cf0154-dd5a-4e12-b08f-95647d9ca4a1","Type":"ContainerDied","Data":"c1b76dea5305f2f8cda5a39ba627bbcaf73e014fc748d7bb0d002bfb3c2be5ee"}
Oct 30 12:59:35.548492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.542863    5837 scope.go:122] "RemoveContainer" containerID="a885069c4414c1032bd18343a335ca32b96c4a0606a7251b5cd909431f8d0593"
Oct 30 12:59:35.548492 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.543034    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"
Oct 30 12:59:35.624586 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.624555    5837 scope.go:122] "RemoveContainer" containerID="a885069c4414c1032bd18343a335ca32b96c4a0606a7251b5cd909431f8d0593"
Oct 30 12:59:35.628196 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:59:35.628157    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a885069c4414c1032bd18343a335ca32b96c4a0606a7251b5cd909431f8d0593\": container with ID starting with a885069c4414c1032bd18343a335ca32b96c4a0606a7251b5cd909431f8d0593 not found: ID does not exist" containerID="a885069c4414c1032bd18343a335ca32b96c4a0606a7251b5cd909431f8d0593"
Oct 30 12:59:35.628449 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.628422    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a885069c4414c1032bd18343a335ca32b96c4a0606a7251b5cd909431f8d0593"} err="failed to get container status \"a885069c4414c1032bd18343a335ca32b96c4a0606a7251b5cd909431f8d0593\": rpc error: code = NotFound desc = could not find container \"a885069c4414c1032bd18343a335ca32b96c4a0606a7251b5cd909431f8d0593\": container with ID starting with a885069c4414c1032bd18343a335ca32b96c4a0606a7251b5cd909431f8d0593 not found: ID does not exist"
Oct 30 12:59:35.645307 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.645274    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"]
Oct 30 12:59:35.652525 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:35.652489    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-c85b58bf5-vjkw9"]
Oct 30 12:59:37.710442 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:37.709960    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1cf0154-dd5a-4e12-b08f-95647d9ca4a1" path="/var/lib/kubelet/pods/f1cf0154-dd5a-4e12-b08f-95647d9ca4a1/volumes"
Oct 30 12:59:38.838369 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:38.838328    5837 patch_prober.go:29] interesting pod/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:59:38.838776 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:38.838443    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4" podUID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:59:40.648356 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.647946    5837 generic.go:356] "Generic (PLEG): container finished" podID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerID="0e80c6fd2e1837884c48db5d58574f1e7254b575d10ce32e2047f0a6318d36f2" exitCode=0
Oct 30 12:59:40.648896 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.648367    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4" event={"ID":"e7c0f7fa-4a94-41ae-a846-8309e089d18e","Type":"ContainerDied","Data":"0e80c6fd2e1837884c48db5d58574f1e7254b575d10ce32e2047f0a6318d36f2"}
Oct 30 12:59:40.648896 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.648466    5837 scope.go:122] "RemoveContainer" containerID="c26c377726944f54e9e85ae5869902e31227aef5e09b97fbe55dddb4cb24c7b4"
Oct 30 12:59:40.700731 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.700703    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:59:40.809409 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.809369    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-data\") pod \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") "
Oct 30 12:59:40.809619 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.809440    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-workload-socket\") pod \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") "
Oct 30 12:59:40.809619 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.809511    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c0f7fa-4a94-41ae-a846-8309e089d18e-kube-api-access-5r6t9\" (UniqueName: \"kubernetes.io/projected/e7c0f7fa-4a94-41ae-a846-8309e089d18e-kube-api-access-5r6t9\") pod \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") "
Oct 30 12:59:40.809619 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.809574    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-workload-certs\") pod \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") "
Oct 30 12:59:40.809806 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.809625    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-token\" (UniqueName: \"kubernetes.io/projected/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-token\") pod \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") "
Oct 30 12:59:40.809806 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.809684    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-data" pod "e7c0f7fa-4a94-41ae-a846-8309e089d18e" (UID: "e7c0f7fa-4a94-41ae-a846-8309e089d18e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:59:40.809806 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.809709    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-credential-socket\") pod \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") "
Oct 30 12:59:40.809806 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.809733    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-workload-socket" pod "e7c0f7fa-4a94-41ae-a846-8309e089d18e" (UID: "e7c0f7fa-4a94-41ae-a846-8309e089d18e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:59:40.809806 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.809787    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-envoy\") pod \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") "
Oct 30 12:59:40.810103 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.809887    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istiod-ca-cert\") pod \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") "
Oct 30 12:59:40.810103 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.809936    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-workload-certs" pod "e7c0f7fa-4a94-41ae-a846-8309e089d18e" (UID: "e7c0f7fa-4a94-41ae-a846-8309e089d18e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:59:40.810103 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.809970    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-podinfo\") pod \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\" (UID: \"e7c0f7fa-4a94-41ae-a846-8309e089d18e\") "
Oct 30 12:59:40.810103 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.810003    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-credential-socket" pod "e7c0f7fa-4a94-41ae-a846-8309e089d18e" (UID: "e7c0f7fa-4a94-41ae-a846-8309e089d18e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:59:40.810809 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.810785    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:40.810942 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.810812    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:40.810942 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.810826    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:40.810942 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.810839    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:40.811579 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.811514    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istiod-ca-cert" pod "e7c0f7fa-4a94-41ae-a846-8309e089d18e" (UID: "e7c0f7fa-4a94-41ae-a846-8309e089d18e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:40.812698 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.812670    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c0f7fa-4a94-41ae-a846-8309e089d18e-kube-api-access-5r6t9" pod "e7c0f7fa-4a94-41ae-a846-8309e089d18e" (UID: "e7c0f7fa-4a94-41ae-a846-8309e089d18e"). InnerVolumeSpecName "kube-api-access-5r6t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:40.813066 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.813042    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-token" pod "e7c0f7fa-4a94-41ae-a846-8309e089d18e" (UID: "e7c0f7fa-4a94-41ae-a846-8309e089d18e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:40.814737 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.814715    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-envoy" pod "e7c0f7fa-4a94-41ae-a846-8309e089d18e" (UID: "e7c0f7fa-4a94-41ae-a846-8309e089d18e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:59:40.814874 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.814810    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-podinfo" pod "e7c0f7fa-4a94-41ae-a846-8309e089d18e" (UID: "e7c0f7fa-4a94-41ae-a846-8309e089d18e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:59:40.912228 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.912161    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r6t9\" (UniqueName: \"kubernetes.io/projected/e7c0f7fa-4a94-41ae-a846-8309e089d18e-kube-api-access-5r6t9\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:40.912228 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.912189    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:40.912228 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.912199    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:40.912228 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.912207    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:40.912228 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:40.912214    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c0f7fa-4a94-41ae-a846-8309e089d18e-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:41.027527 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:41.027496    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"]
Oct 30 12:59:41.030814 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:41.030790    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e47c1373-llcp5"]
Oct 30 12:59:41.610066 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:41.610037    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7e0add5-e41c-43c6-9557-f21f19e4ff3a" path="/var/lib/kubelet/pods/b7e0add5-e41c-43c6-9557-f21f19e4ff3a/volumes"
Oct 30 12:59:41.658345 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:41.658323    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"
Oct 30 12:59:41.658758 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:41.658361    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4" event={"ID":"e7c0f7fa-4a94-41ae-a846-8309e089d18e","Type":"ContainerDied","Data":"5da72d611980f57d9d0331b7a2843910372b1677a48f0f1947cf4acc71cb1a06"}
Oct 30 12:59:41.658758 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:41.658425    5837 scope.go:122] "RemoveContainer" containerID="0e80c6fd2e1837884c48db5d58574f1e7254b575d10ce32e2047f0a6318d36f2"
Oct 30 12:59:41.720951 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:41.720924    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"]
Oct 30 12:59:41.724251 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:41.724227    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-70621c-openshift-default-74bc85477-kvzf4"]
Oct 30 12:59:42.608421 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:42.602440    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"]
Oct 30 12:59:42.616465 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:42.612211    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"]
Oct 30 12:59:43.609811 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:43.609775    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" path="/var/lib/kubelet/pods/e7c0f7fa-4a94-41ae-a846-8309e089d18e/volumes"
Oct 30 12:59:44.516420 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:44.516230    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"
Oct 30 12:59:44.528458 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:44.517427    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"
Oct 30 12:59:45.128246 ip-10-0-137-228 kubenswrapper[5837]: E1030 12:59:45.128215    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 12:59:46.630502 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.630189    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"]
Oct 30 12:59:46.631375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.631335    5837 cpu_manager.go:414] "containerMap: removing container" podUID="f1cf0154-dd5a-4e12-b08f-95647d9ca4a1" containerName="payload-pre-processing"
Oct 30 12:59:46.631375 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.631363    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1cf0154-dd5a-4e12-b08f-95647d9ca4a1" containerName="payload-pre-processing"
Oct 30 12:59:46.631658 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.631436    5837 cpu_manager.go:414] "containerMap: removing container" podUID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerName="istio-proxy"
Oct 30 12:59:46.631658 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.631445    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerName="istio-proxy"
Oct 30 12:59:46.631658 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.631493    5837 cpu_manager.go:414] "containerMap: removing container" podUID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerName="istio-proxy"
Oct 30 12:59:46.631658 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.631501    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerName="istio-proxy"
Oct 30 12:59:46.631877 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.631776    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerName="istio-proxy"
Oct 30 12:59:46.631877 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.631805    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c0f7fa-4a94-41ae-a846-8309e089d18e" containerName="istio-proxy"
Oct 30 12:59:46.631877 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.631820    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1cf0154-dd5a-4e12-b08f-95647d9ca4a1" containerName="payload-pre-processing"
Oct 30 12:59:46.637625 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.637600    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"
Oct 30 12:59:46.655557 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.655517    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"]
Oct 30 12:59:46.660866 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.660844    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f0a01c-dockercfg-5rjrs\""
Oct 30 12:59:46.661006 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.660835    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f0a01c\""
Oct 30 12:59:46.782364 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.782143    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed765745-1281-4261-a959-0695a04713b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff\" (UID: \"ed765745-1281-4261-a959-0695a04713b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"
Oct 30 12:59:46.854213 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.854171    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq59x\" (UniqueName: \"kubernetes.io/projected/ed765745-1281-4261-a959-0695a04713b1-kube-api-access-mq59x\") pod \"payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff\" (UID: \"ed765745-1281-4261-a959-0695a04713b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"
Oct 30 12:59:46.861383 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.861354    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"]
Oct 30 12:59:46.869594 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.869485    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"]
Oct 30 12:59:46.870728 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.870704    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"
Oct 30 12:59:46.956239 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.956155    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed765745-1281-4261-a959-0695a04713b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff\" (UID: \"ed765745-1281-4261-a959-0695a04713b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"
Oct 30 12:59:46.956239 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.956225    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntk4p\" (UniqueName: \"kubernetes.io/projected/6e913224-538a-4f4a-ba4f-c87d3446cb96-kube-api-access-ntk4p\") pod \"payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p\" (UID: \"6e913224-538a-4f4a-ba4f-c87d3446cb96\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"
Oct 30 12:59:46.956548 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.956253    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e913224-538a-4f4a-ba4f-c87d3446cb96-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p\" (UID: \"6e913224-538a-4f4a-ba4f-c87d3446cb96\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"
Oct 30 12:59:46.956548 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.956455    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq59x\" (UniqueName: \"kubernetes.io/projected/ed765745-1281-4261-a959-0695a04713b1-kube-api-access-mq59x\") pod \"payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff\" (UID: \"ed765745-1281-4261-a959-0695a04713b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"
Oct 30 12:59:46.956833 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.956815    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed765745-1281-4261-a959-0695a04713b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff\" (UID: \"ed765745-1281-4261-a959-0695a04713b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"
Oct 30 12:59:46.975450 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:46.975422    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq59x\" (UniqueName: \"kubernetes.io/projected/ed765745-1281-4261-a959-0695a04713b1-kube-api-access-mq59x\") pod \"payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff\" (UID: \"ed765745-1281-4261-a959-0695a04713b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"
Oct 30 12:59:47.057467 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.057432    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntk4p\" (UniqueName: \"kubernetes.io/projected/6e913224-538a-4f4a-ba4f-c87d3446cb96-kube-api-access-ntk4p\") pod \"payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p\" (UID: \"6e913224-538a-4f4a-ba4f-c87d3446cb96\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"
Oct 30 12:59:47.057622 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.057486    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e913224-538a-4f4a-ba4f-c87d3446cb96-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p\" (UID: \"6e913224-538a-4f4a-ba4f-c87d3446cb96\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"
Oct 30 12:59:47.058118 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.058099    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e913224-538a-4f4a-ba4f-c87d3446cb96-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p\" (UID: \"6e913224-538a-4f4a-ba4f-c87d3446cb96\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"
Oct 30 12:59:47.077019 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.076994    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntk4p\" (UniqueName: \"kubernetes.io/projected/6e913224-538a-4f4a-ba4f-c87d3446cb96-kube-api-access-ntk4p\") pod \"payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p\" (UID: \"6e913224-538a-4f4a-ba4f-c87d3446cb96\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"
Oct 30 12:59:47.191025 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.190939    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"
Oct 30 12:59:47.275496 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.269550    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"
Oct 30 12:59:47.514314 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.514274    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"]
Oct 30 12:59:47.603049 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.602994    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg" podUID="c69d2915-b9d6-45e1-ba5e-60395ce8c199" containerName="payload-processing" containerID="cri-o://0df2dcac8d30edc6212e4912a70c515e7405f98c5d64f935ab1b84271d5b88a9" gracePeriod=25
Oct 30 12:59:47.617111 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.617077    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8" podUID="d6b5d322-d077-443e-88cd-54067bfb8e15" containerName="payload-pre-processing" containerID="cri-o://ca8e91cc987f620aab5d0f846dc84f5d5f85093677e12fa02b9d5e576e32cc4f" gracePeriod=25
Oct 30 12:59:47.623009 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.622984    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"]
Oct 30 12:59:47.670062 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:59:47.670032    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded765745_1281_4261_a959_0695a04713b1.slice/crio-b37a490144a41b9c112f30f8dcb01cfd9d9663e615d3b5fd759f4ed668d643f4 WatchSource:0}: Error finding container b37a490144a41b9c112f30f8dcb01cfd9d9663e615d3b5fd759f4ed668d643f4: Status 404 returned error can't find the container with id b37a490144a41b9c112f30f8dcb01cfd9d9663e615d3b5fd759f4ed668d643f4
Oct 30 12:59:47.879720 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.879681    5837 generic.go:356] "Generic (PLEG): container finished" podID="c69d2915-b9d6-45e1-ba5e-60395ce8c199" containerID="0df2dcac8d30edc6212e4912a70c515e7405f98c5d64f935ab1b84271d5b88a9" exitCode=0
Oct 30 12:59:47.879902 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.879808    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg" event={"ID":"c69d2915-b9d6-45e1-ba5e-60395ce8c199","Type":"ContainerDied","Data":"0df2dcac8d30edc6212e4912a70c515e7405f98c5d64f935ab1b84271d5b88a9"}
Oct 30 12:59:47.881731 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.881700    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff" event={"ID":"ed765745-1281-4261-a959-0695a04713b1","Type":"ContainerStarted","Data":"68279432dc3cab35abe8114994ce01db0a1e157923caa8a771d70eaa5ffdcee6"}
Oct 30 12:59:47.881875 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.881756    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff" event={"ID":"ed765745-1281-4261-a959-0695a04713b1","Type":"ContainerStarted","Data":"b37a490144a41b9c112f30f8dcb01cfd9d9663e615d3b5fd759f4ed668d643f4"}
Oct 30 12:59:47.882872 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.882566    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"
Oct 30 12:59:47.885125 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.885106    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p" event={"ID":"6e913224-538a-4f4a-ba4f-c87d3446cb96","Type":"ContainerStarted","Data":"e8449940fcdc1258abffa00e7c651103d5c5b8928a1e70d4bff53c7ce804fc86"}
Oct 30 12:59:47.885281 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.885267    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p" event={"ID":"6e913224-538a-4f4a-ba4f-c87d3446cb96","Type":"ContainerStarted","Data":"b2ab1a277906483bd046a39b8d70095aa6ed2101e94516818e949e2e12274593"}
Oct 30 12:59:47.886085 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.886069    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"
Oct 30 12:59:47.888083 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.888064    5837 generic.go:356] "Generic (PLEG): container finished" podID="d6b5d322-d077-443e-88cd-54067bfb8e15" containerID="ca8e91cc987f620aab5d0f846dc84f5d5f85093677e12fa02b9d5e576e32cc4f" exitCode=0
Oct 30 12:59:47.888260 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.888193    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8" event={"ID":"d6b5d322-d077-443e-88cd-54067bfb8e15","Type":"ContainerDied","Data":"ca8e91cc987f620aab5d0f846dc84f5d5f85093677e12fa02b9d5e576e32cc4f"}
Oct 30 12:59:47.929973 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.929945    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"
Oct 30 12:59:47.934782 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.933382    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff" podStartSLOduration=1.933369802 podStartE2EDuration="1.933369802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:47.929525464 +0000 UTC m=+1376.453809683" watchObservedRunningTime="2026-10-30 12:59:47.933369802 +0000 UTC m=+1376.457654020"
Oct 30 12:59:47.958078 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.958051    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"
Oct 30 12:59:47.978581 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:47.978527    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p" podStartSLOduration=1.978509668 podStartE2EDuration="1.978509668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:47.972709436 +0000 UTC m=+1376.496993655" watchObservedRunningTime="2026-10-30 12:59:47.978509668 +0000 UTC m=+1376.502793890"
Oct 30 12:59:48.004970 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.004944    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b5d322-d077-443e-88cd-54067bfb8e15-kube-api-access-9z5vs\" (UniqueName: \"kubernetes.io/projected/d6b5d322-d077-443e-88cd-54067bfb8e15-kube-api-access-9z5vs\") pod \"d6b5d322-d077-443e-88cd-54067bfb8e15\" (UID: \"d6b5d322-d077-443e-88cd-54067bfb8e15\") "
Oct 30 12:59:48.005140 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.005121    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b5d322-d077-443e-88cd-54067bfb8e15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b5d322-d077-443e-88cd-54067bfb8e15-plugins-config-volume\") pod \"d6b5d322-d077-443e-88cd-54067bfb8e15\" (UID: \"d6b5d322-d077-443e-88cd-54067bfb8e15\") "
Oct 30 12:59:48.005272 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.005233    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c69d2915-b9d6-45e1-ba5e-60395ce8c199-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69d2915-b9d6-45e1-ba5e-60395ce8c199-plugins-config-volume\") pod \"c69d2915-b9d6-45e1-ba5e-60395ce8c199\" (UID: \"c69d2915-b9d6-45e1-ba5e-60395ce8c199\") "
Oct 30 12:59:48.005345 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.005327    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c69d2915-b9d6-45e1-ba5e-60395ce8c199-kube-api-access-6nv8l\" (UniqueName: \"kubernetes.io/projected/c69d2915-b9d6-45e1-ba5e-60395ce8c199-kube-api-access-6nv8l\") pod \"c69d2915-b9d6-45e1-ba5e-60395ce8c199\" (UID: \"c69d2915-b9d6-45e1-ba5e-60395ce8c199\") "
Oct 30 12:59:48.005619 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.005596    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b5d322-d077-443e-88cd-54067bfb8e15-plugins-config-volume" pod "d6b5d322-d077-443e-88cd-54067bfb8e15" (UID: "d6b5d322-d077-443e-88cd-54067bfb8e15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:48.005713 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.005647    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69d2915-b9d6-45e1-ba5e-60395ce8c199-plugins-config-volume" pod "c69d2915-b9d6-45e1-ba5e-60395ce8c199" (UID: "c69d2915-b9d6-45e1-ba5e-60395ce8c199"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:48.006036 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.006019    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b5d322-d077-443e-88cd-54067bfb8e15-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:48.006145 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.006129    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69d2915-b9d6-45e1-ba5e-60395ce8c199-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:48.008188 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.008165    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b5d322-d077-443e-88cd-54067bfb8e15-kube-api-access-9z5vs" pod "d6b5d322-d077-443e-88cd-54067bfb8e15" (UID: "d6b5d322-d077-443e-88cd-54067bfb8e15"). InnerVolumeSpecName "kube-api-access-9z5vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:48.008270 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.008228    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69d2915-b9d6-45e1-ba5e-60395ce8c199-kube-api-access-6nv8l" pod "c69d2915-b9d6-45e1-ba5e-60395ce8c199" (UID: "c69d2915-b9d6-45e1-ba5e-60395ce8c199"). InnerVolumeSpecName "kube-api-access-6nv8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:48.116414 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.115533    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z5vs\" (UniqueName: \"kubernetes.io/projected/d6b5d322-d077-443e-88cd-54067bfb8e15-kube-api-access-9z5vs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:48.116414 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.115581    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nv8l\" (UniqueName: \"kubernetes.io/projected/c69d2915-b9d6-45e1-ba5e-60395ce8c199-kube-api-access-6nv8l\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 12:59:48.897374 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.897343    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"
Oct 30 12:59:48.897861 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.897376    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8" event={"ID":"d6b5d322-d077-443e-88cd-54067bfb8e15","Type":"ContainerDied","Data":"46188f640906e323c8b225acda978da69b301533552f74075dfa66788f184ff5"}
Oct 30 12:59:48.897861 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.897458    5837 scope.go:122] "RemoveContainer" containerID="ca8e91cc987f620aab5d0f846dc84f5d5f85093677e12fa02b9d5e576e32cc4f"
Oct 30 12:59:48.899219 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.899197    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg" event={"ID":"c69d2915-b9d6-45e1-ba5e-60395ce8c199","Type":"ContainerDied","Data":"97b4e82e0259b9ef2904e564621e5d4f8d2b78eee7e5df67f032de944247f3f6"}
Oct 30 12:59:48.899318 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.899204    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"
Oct 30 12:59:48.911285 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.911264    5837 scope.go:122] "RemoveContainer" containerID="0df2dcac8d30edc6212e4912a70c515e7405f98c5d64f935ab1b84271d5b88a9"
Oct 30 12:59:48.950206 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.950175    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"]
Oct 30 12:59:48.961195 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:48.961175    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-5fd75db559-qhvk8"]
Oct 30 12:59:49.003513 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:49.003487    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"]
Oct 30 12:59:49.008723 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:49.008703    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f1222e-7f85759464-x48hg"]
Oct 30 12:59:49.606164 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:49.606133    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c69d2915-b9d6-45e1-ba5e-60395ce8c199" path="/var/lib/kubelet/pods/c69d2915-b9d6-45e1-ba5e-60395ce8c199/volumes"
Oct 30 12:59:49.606596 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:49.606583    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b5d322-d077-443e-88cd-54067bfb8e15" path="/var/lib/kubelet/pods/d6b5d322-d077-443e-88cd-54067bfb8e15/volumes"
Oct 30 12:59:56.160708 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.160675    5837 scope.go:122] "RemoveContainer" containerID="9e46a059e169bc4f7501fba51284d17fc91342d4dcb84508ed36bb9c4ad416f6"
Oct 30 12:59:56.174065 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.174037    5837 scope.go:122] "RemoveContainer" containerID="202bbed235e000aa98d097d5d918d1c4400811e2239604d541d1b4d49c974842"
Oct 30 12:59:56.187785 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.187761    5837 scope.go:122] "RemoveContainer" containerID="f742bf746fec2f5d39068b088225e06c3f29d8331135f9b0c7306b656b8198b5"
Oct 30 12:59:56.197494 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.197473    5837 scope.go:122] "RemoveContainer" containerID="f3e75fcb18b0116e5ad2cc8d8622314b02c61a450075da6cf4889e02883be67e"
Oct 30 12:59:56.206868 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.206849    5837 scope.go:122] "RemoveContainer" containerID="abf4939e5998dbe659db1606f09c5d4a1df4ebc41991bb972fd73abbd0f766a6"
Oct 30 12:59:56.216200 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.216180    5837 scope.go:122] "RemoveContainer" containerID="750487f73b3567f9a1cca9c6b8899719c37596c8f3ff6e80ff46fe85d608d404"
Oct 30 12:59:56.225263 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.225247    5837 scope.go:122] "RemoveContainer" containerID="768613622ff22d520b5f7d268babd930840d2541020e674d253530cff8c7fb6c"
Oct 30 12:59:56.234499 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.234481    5837 scope.go:122] "RemoveContainer" containerID="89323964a29e56d585a3928ec4c2921558117221e541a0f10ea62c445b50f81f"
Oct 30 12:59:56.247325 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.247300    5837 scope.go:122] "RemoveContainer" containerID="af06b31008684dc78cb39f46122b9644f3da5b764b3d36502b4f712f0cf54a58"
Oct 30 12:59:56.266334 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.266308    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"]
Oct 30 12:59:56.267065 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.267046    5837 cpu_manager.go:414] "containerMap: removing container" podUID="c69d2915-b9d6-45e1-ba5e-60395ce8c199" containerName="payload-processing"
Oct 30 12:59:56.267162 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.267066    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c69d2915-b9d6-45e1-ba5e-60395ce8c199" containerName="payload-processing"
Oct 30 12:59:56.267162 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.267122    5837 cpu_manager.go:414] "containerMap: removing container" podUID="d6b5d322-d077-443e-88cd-54067bfb8e15" containerName="payload-pre-processing"
Oct 30 12:59:56.267162 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.267131    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b5d322-d077-443e-88cd-54067bfb8e15" containerName="payload-pre-processing"
Oct 30 12:59:56.267334 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.267320    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="c69d2915-b9d6-45e1-ba5e-60395ce8c199" containerName="payload-processing"
Oct 30 12:59:56.267432 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.267350    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b5d322-d077-443e-88cd-54067bfb8e15" containerName="payload-pre-processing"
Oct 30 12:59:56.273811 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.273713    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"
Oct 30 12:59:56.292371 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.292353    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-096cc3\""
Oct 30 12:59:56.303800 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.303776    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-096cc3-dockercfg-s7cft\""
Oct 30 12:59:56.309296 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.309270    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"]
Oct 30 12:59:56.328681 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.328657    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"]
Oct 30 12:59:56.353521 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.353418    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"
Oct 30 12:59:56.354878 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.354732    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rjk8\" (UniqueName: \"kubernetes.io/projected/9abf7013-9e1f-475c-9fb9-9e928a7dcb79-kube-api-access-7rjk8\") pod \"payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz\" (UID: \"9abf7013-9e1f-475c-9fb9-9e928a7dcb79\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"
Oct 30 12:59:56.355139 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.355118    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abf7013-9e1f-475c-9fb9-9e928a7dcb79-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz\" (UID: \"9abf7013-9e1f-475c-9fb9-9e928a7dcb79\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"
Oct 30 12:59:56.521425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.519825    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53dd4fc3-081f-4b41-bedc-d6b87a44aec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk\" (UID: \"53dd4fc3-081f-4b41-bedc-d6b87a44aec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"
Oct 30 12:59:56.521425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.520006    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abf7013-9e1f-475c-9fb9-9e928a7dcb79-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz\" (UID: \"9abf7013-9e1f-475c-9fb9-9e928a7dcb79\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"
Oct 30 12:59:56.521425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.520081    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rjk8\" (UniqueName: \"kubernetes.io/projected/9abf7013-9e1f-475c-9fb9-9e928a7dcb79-kube-api-access-7rjk8\") pod \"payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz\" (UID: \"9abf7013-9e1f-475c-9fb9-9e928a7dcb79\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"
Oct 30 12:59:56.521425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.520171    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4h8v\" (UniqueName: \"kubernetes.io/projected/53dd4fc3-081f-4b41-bedc-d6b87a44aec3-kube-api-access-x4h8v\") pod \"payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk\" (UID: \"53dd4fc3-081f-4b41-bedc-d6b87a44aec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"
Oct 30 12:59:56.521425 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.521361    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abf7013-9e1f-475c-9fb9-9e928a7dcb79-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz\" (UID: \"9abf7013-9e1f-475c-9fb9-9e928a7dcb79\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"
Oct 30 12:59:56.586666 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.555454    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"]
Oct 30 12:59:56.622536 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.622268    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rjk8\" (UniqueName: \"kubernetes.io/projected/9abf7013-9e1f-475c-9fb9-9e928a7dcb79-kube-api-access-7rjk8\") pod \"payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz\" (UID: \"9abf7013-9e1f-475c-9fb9-9e928a7dcb79\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"
Oct 30 12:59:56.625837 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.624921    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"
Oct 30 12:59:56.630016 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.629976    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53dd4fc3-081f-4b41-bedc-d6b87a44aec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk\" (UID: \"53dd4fc3-081f-4b41-bedc-d6b87a44aec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"
Oct 30 12:59:56.630175 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.630144    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4h8v\" (UniqueName: \"kubernetes.io/projected/53dd4fc3-081f-4b41-bedc-d6b87a44aec3-kube-api-access-x4h8v\") pod \"payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk\" (UID: \"53dd4fc3-081f-4b41-bedc-d6b87a44aec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"
Oct 30 12:59:56.631339 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.631312    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53dd4fc3-081f-4b41-bedc-d6b87a44aec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk\" (UID: \"53dd4fc3-081f-4b41-bedc-d6b87a44aec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"
Oct 30 12:59:56.650359 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.650337    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4h8v\" (UniqueName: \"kubernetes.io/projected/53dd4fc3-081f-4b41-bedc-d6b87a44aec3-kube-api-access-x4h8v\") pod \"payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk\" (UID: \"53dd4fc3-081f-4b41-bedc-d6b87a44aec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"
Oct 30 12:59:56.840455 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.840374    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"
Oct 30 12:59:56.963325 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:56.963295    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"]
Oct 30 12:59:56.965949 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:59:56.965921    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abf7013_9e1f_475c_9fb9_9e928a7dcb79.slice/crio-76aa59d4ef26d9cb3296065aafd55c0f0d113bfbdd17b6a84bed36e61e237cd0 WatchSource:0}: Error finding container 76aa59d4ef26d9cb3296065aafd55c0f0d113bfbdd17b6a84bed36e61e237cd0: Status 404 returned error can't find the container with id 76aa59d4ef26d9cb3296065aafd55c0f0d113bfbdd17b6a84bed36e61e237cd0
Oct 30 12:59:57.051989 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:57.051961    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz" event={"ID":"9abf7013-9e1f-475c-9fb9-9e928a7dcb79","Type":"ContainerStarted","Data":"76aa59d4ef26d9cb3296065aafd55c0f0d113bfbdd17b6a84bed36e61e237cd0"}
Oct 30 12:59:57.149262 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:57.149237    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"]
Oct 30 12:59:57.152039 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:59:57.152009    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53dd4fc3_081f_4b41_bedc_d6b87a44aec3.slice/crio-633cb6060bbe0f6a844979aa37d4b4fc55a9c970925db9e31b5a4601f4a6e271 WatchSource:0}: Error finding container 633cb6060bbe0f6a844979aa37d4b4fc55a9c970925db9e31b5a4601f4a6e271: Status 404 returned error can't find the container with id 633cb6060bbe0f6a844979aa37d4b4fc55a9c970925db9e31b5a4601f4a6e271
Oct 30 12:59:58.061263 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:58.061230    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk" event={"ID":"53dd4fc3-081f-4b41-bedc-d6b87a44aec3","Type":"ContainerStarted","Data":"a1b13d18d66972a525913da81918d8efc4f0ea88aa5e9737b632e921f882e4c7"}
Oct 30 12:59:58.061744 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:58.061286    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk" event={"ID":"53dd4fc3-081f-4b41-bedc-d6b87a44aec3","Type":"ContainerStarted","Data":"633cb6060bbe0f6a844979aa37d4b4fc55a9c970925db9e31b5a4601f4a6e271"}
Oct 30 12:59:58.061744 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:58.061328    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"
Oct 30 12:59:58.062942 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:58.062923    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz" event={"ID":"9abf7013-9e1f-475c-9fb9-9e928a7dcb79","Type":"ContainerStarted","Data":"e205a7a0b86b8ea174fc73571c2ab4d94b5cf43aa6991bd4bbfea99656cdc085"}
Oct 30 12:59:58.063063 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:58.063048    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"
Oct 30 12:59:58.101139 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:58.101089    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk" podStartSLOduration=2.101072789 podStartE2EDuration="2.101072789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:58.098767232 +0000 UTC m=+1386.623051451" watchObservedRunningTime="2026-10-30 12:59:58.101072789 +0000 UTC m=+1386.625357008"
Oct 30 12:59:58.139377 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:58.139327    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz" podStartSLOduration=2.139309826 podStartE2EDuration="2.139309826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:58.137653097 +0000 UTC m=+1386.661937317" watchObservedRunningTime="2026-10-30 12:59:58.139309826 +0000 UTC m=+1386.663594046"
Oct 30 12:59:58.903300 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:58.903272    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"
Oct 30 12:59:59.236875 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.236847    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"]
Oct 30 12:59:59.248771 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.248747    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"]
Oct 30 12:59:59.249006 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.248992    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"]
Oct 30 12:59:59.249175 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.248876    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"
Oct 30 12:59:59.255662 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.255642    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"]
Oct 30 12:59:59.255847 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.255766    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"
Oct 30 12:59:59.263795 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.263760    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f1222e\""
Oct 30 12:59:59.263923 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.263837    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f1222e-dockercfg-9664r\""
Oct 30 12:59:59.308351 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.308314    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86d0755-fbd0-4714-9572-3612ed5198b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj\" (UID: \"d86d0755-fbd0-4714-9572-3612ed5198b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"
Oct 30 12:59:59.308578 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.308378    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c25fff-6919-42ac-9455-f15aa3988883-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw\" (UID: \"85c25fff-6919-42ac-9455-f15aa3988883\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"
Oct 30 12:59:59.308578 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.308437    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zht7j\" (UniqueName: \"kubernetes.io/projected/85c25fff-6919-42ac-9455-f15aa3988883-kube-api-access-zht7j\") pod \"payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw\" (UID: \"85c25fff-6919-42ac-9455-f15aa3988883\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"
Oct 30 12:59:59.308704 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.308627    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m228t\" (UniqueName: \"kubernetes.io/projected/d86d0755-fbd0-4714-9572-3612ed5198b2-kube-api-access-m228t\") pod \"payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj\" (UID: \"d86d0755-fbd0-4714-9572-3612ed5198b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"
Oct 30 12:59:59.410540 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.410508    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86d0755-fbd0-4714-9572-3612ed5198b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj\" (UID: \"d86d0755-fbd0-4714-9572-3612ed5198b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"
Oct 30 12:59:59.410709 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.410564    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c25fff-6919-42ac-9455-f15aa3988883-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw\" (UID: \"85c25fff-6919-42ac-9455-f15aa3988883\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"
Oct 30 12:59:59.410709 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.410589    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zht7j\" (UniqueName: \"kubernetes.io/projected/85c25fff-6919-42ac-9455-f15aa3988883-kube-api-access-zht7j\") pod \"payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw\" (UID: \"85c25fff-6919-42ac-9455-f15aa3988883\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"
Oct 30 12:59:59.410709 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.410626    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m228t\" (UniqueName: \"kubernetes.io/projected/d86d0755-fbd0-4714-9572-3612ed5198b2-kube-api-access-m228t\") pod \"payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj\" (UID: \"d86d0755-fbd0-4714-9572-3612ed5198b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"
Oct 30 12:59:59.411285 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.411267    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c25fff-6919-42ac-9455-f15aa3988883-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw\" (UID: \"85c25fff-6919-42ac-9455-f15aa3988883\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"
Oct 30 12:59:59.411322 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.411282    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86d0755-fbd0-4714-9572-3612ed5198b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj\" (UID: \"d86d0755-fbd0-4714-9572-3612ed5198b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"
Oct 30 12:59:59.429439 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.429381    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zht7j\" (UniqueName: \"kubernetes.io/projected/85c25fff-6919-42ac-9455-f15aa3988883-kube-api-access-zht7j\") pod \"payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw\" (UID: \"85c25fff-6919-42ac-9455-f15aa3988883\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"
Oct 30 12:59:59.429602 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.429520    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m228t\" (UniqueName: \"kubernetes.io/projected/d86d0755-fbd0-4714-9572-3612ed5198b2-kube-api-access-m228t\") pod \"payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj\" (UID: \"d86d0755-fbd0-4714-9572-3612ed5198b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"
Oct 30 12:59:59.570327 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.570255    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"
Oct 30 12:59:59.582587 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.582564    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"
Oct 30 12:59:59.896975 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.896944    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"]
Oct 30 12:59:59.897527 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:59:59.897498    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c25fff_6919_42ac_9455_f15aa3988883.slice/crio-214f612afffe3e80f9e662f5ce30c2dd20aaff07359bcd6b3cfddd6d4d89574a WatchSource:0}: Error finding container 214f612afffe3e80f9e662f5ce30c2dd20aaff07359bcd6b3cfddd6d4d89574a: Status 404 returned error can't find the container with id 214f612afffe3e80f9e662f5ce30c2dd20aaff07359bcd6b3cfddd6d4d89574a
Oct 30 12:59:59.914560 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.914540    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"
Oct 30 12:59:59.927649 ip-10-0-137-228 kubenswrapper[5837]: I1030 12:59:59.927622    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"]
Oct 30 12:59:59.933684 ip-10-0-137-228 kubenswrapper[5837]: W1030 12:59:59.933662    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86d0755_fbd0_4714_9572_3612ed5198b2.slice/crio-70c89145770aa97c306d338bb7f7d9a64debff5fd34f5beed7081d1fac33e714 WatchSource:0}: Error finding container 70c89145770aa97c306d338bb7f7d9a64debff5fd34f5beed7081d1fac33e714: Status 404 returned error can't find the container with id 70c89145770aa97c306d338bb7f7d9a64debff5fd34f5beed7081d1fac33e714
Oct 30 13:00:00.145094 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.145005    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"]
Oct 30 13:00:00.150122 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.150097    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"]
Oct 30 13:00:00.150286 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.150221    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"
Oct 30 13:00:00.154527 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.154508    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"]
Oct 30 13:00:00.154657 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.154640    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"
Oct 30 13:00:00.159100 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.159080    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"]
Oct 30 13:00:00.159224 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.159177    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"
Oct 30 13:00:00.171711 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.167590    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"]
Oct 30 13:00:00.171711 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.167688    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"
Oct 30 13:00:00.171711 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.168978    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:00:00.171711 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.169750    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:00:00.177065 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.177040    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"]
Oct 30 13:00:00.188442 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.181844    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"]
Oct 30 13:00:00.196121 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.195354    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"]
Oct 30 13:00:00.196121 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.195457    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"]
Oct 30 13:00:00.196754 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.196235    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"
Oct 30 13:00:00.210739 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.208502    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"]
Oct 30 13:00:00.210739 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.208606    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"
Oct 30 13:00:00.217555 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.217228    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"]
Oct 30 13:00:00.236644 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.236621    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"]
Oct 30 13:00:00.237828 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.237789    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6krb4\" (UniqueName: \"kubernetes.io/projected/84fb8f80-260a-41ee-9df6-3a7f5a21a9df-kube-api-access-6krb4\") pod \"maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27\" (UID: \"84fb8f80-260a-41ee-9df6-3a7f5a21a9df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"
Oct 30 13:00:00.238241 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.237990    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fb8f80-260a-41ee-9df6-3a7f5a21a9df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27\" (UID: \"84fb8f80-260a-41ee-9df6-3a7f5a21a9df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"
Oct 30 13:00:00.251110 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.251086    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:00:00.254635 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.254610    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj" event={"ID":"d86d0755-fbd0-4714-9572-3612ed5198b2","Type":"ContainerStarted","Data":"0d5af89b5bfd805952f47d149f2c48fba1eb1bc4fe42e1e43ea602eaf6c8c291"}
Oct 30 13:00:00.254801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.254786    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj" event={"ID":"d86d0755-fbd0-4714-9572-3612ed5198b2","Type":"ContainerStarted","Data":"70c89145770aa97c306d338bb7f7d9a64debff5fd34f5beed7081d1fac33e714"}
Oct 30 13:00:00.255001 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.254958    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"
Oct 30 13:00:00.256806 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.256780    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw" event={"ID":"85c25fff-6919-42ac-9455-f15aa3988883","Type":"ContainerStarted","Data":"f7fe027a22db1c01db4142024e438df2f2813e426d14ad1d71e572a662045bcb"}
Oct 30 13:00:00.256928 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.256830    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw" event={"ID":"85c25fff-6919-42ac-9455-f15aa3988883","Type":"ContainerStarted","Data":"214f612afffe3e80f9e662f5ce30c2dd20aaff07359bcd6b3cfddd6d4d89574a"}
Oct 30 13:00:00.256928 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.256875    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"
Oct 30 13:00:00.341351 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.341313    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71007e75-4683-4704-bf4e-11cd7669f1ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz\" (UID: \"71007e75-4683-4704-bf4e-11cd7669f1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"
Oct 30 13:00:00.341674 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.341642    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bb0c30-e293-4b79-9e36-d37599110835-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95\" (UID: \"b6bb0c30-e293-4b79-9e36-d37599110835\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"
Oct 30 13:00:00.341831 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.341733    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e00d30-b982-4c55-b2a5-984eb00676ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf\" (UID: \"e2e00d30-b982-4c55-b2a5-984eb00676ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"
Oct 30 13:00:00.341831 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.341777    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjpbq\" (UniqueName: \"kubernetes.io/projected/2f81fbc0-f46e-4730-8e0d-d44a67f089c8-kube-api-access-xjpbq\") pod \"maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8\" (UID: \"2f81fbc0-f46e-4730-8e0d-d44a67f089c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"
Oct 30 13:00:00.341916 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.341831    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f81fbc0-f46e-4730-8e0d-d44a67f089c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8\" (UID: \"2f81fbc0-f46e-4730-8e0d-d44a67f089c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"
Oct 30 13:00:00.341916 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.341883    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fb8f80-260a-41ee-9df6-3a7f5a21a9df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27\" (UID: \"84fb8f80-260a-41ee-9df6-3a7f5a21a9df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"
Oct 30 13:00:00.341916 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.341911    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xtzn\" (UniqueName: \"kubernetes.io/projected/71007e75-4683-4704-bf4e-11cd7669f1ba-kube-api-access-5xtzn\") pod \"maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz\" (UID: \"71007e75-4683-4704-bf4e-11cd7669f1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"
Oct 30 13:00:00.342115 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.342096    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv6pt\" (UniqueName: \"kubernetes.io/projected/0fa90322-42c2-45b4-a25a-2d4b68c45164-kube-api-access-bv6pt\") pod \"maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w\" (UID: \"0fa90322-42c2-45b4-a25a-2d4b68c45164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"
Oct 30 13:00:00.342166 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.342133    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gllsr\" (UniqueName: \"kubernetes.io/projected/e2e00d30-b982-4c55-b2a5-984eb00676ef-kube-api-access-gllsr\") pod \"maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf\" (UID: \"e2e00d30-b982-4c55-b2a5-984eb00676ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"
Oct 30 13:00:00.342219 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.342185    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kfbr\" (UniqueName: \"kubernetes.io/projected/b6bb0c30-e293-4b79-9e36-d37599110835-kube-api-access-8kfbr\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95\" (UID: \"b6bb0c30-e293-4b79-9e36-d37599110835\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"
Oct 30 13:00:00.342261 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.342218    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa90322-42c2-45b4-a25a-2d4b68c45164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w\" (UID: \"0fa90322-42c2-45b4-a25a-2d4b68c45164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"
Oct 30 13:00:00.342261 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.342256    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6krb4\" (UniqueName: \"kubernetes.io/projected/84fb8f80-260a-41ee-9df6-3a7f5a21a9df-kube-api-access-6krb4\") pod \"maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27\" (UID: \"84fb8f80-260a-41ee-9df6-3a7f5a21a9df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"
Oct 30 13:00:00.342665 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.342626    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj" podStartSLOduration=1.342614417 podStartE2EDuration="1.342614417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:00.339353093 +0000 UTC m=+1388.863637335" watchObservedRunningTime="2026-10-30 13:00:00.342614417 +0000 UTC m=+1388.866898636"
Oct 30 13:00:00.343053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.343036    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fb8f80-260a-41ee-9df6-3a7f5a21a9df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27\" (UID: \"84fb8f80-260a-41ee-9df6-3a7f5a21a9df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"
Oct 30 13:00:00.361342 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.361299    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6krb4\" (UniqueName: \"kubernetes.io/projected/84fb8f80-260a-41ee-9df6-3a7f5a21a9df-kube-api-access-6krb4\") pod \"maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27\" (UID: \"84fb8f80-260a-41ee-9df6-3a7f5a21a9df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"
Oct 30 13:00:00.414063 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.413949    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw" podStartSLOduration=2.413930693 podStartE2EDuration="2.413930693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:00.41202298 +0000 UTC m=+1388.936307199" watchObservedRunningTime="2026-10-30 13:00:00.413930693 +0000 UTC m=+1388.938214908"
Oct 30 13:00:00.443378 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.443347    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv6pt\" (UniqueName: \"kubernetes.io/projected/0fa90322-42c2-45b4-a25a-2d4b68c45164-kube-api-access-bv6pt\") pod \"maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w\" (UID: \"0fa90322-42c2-45b4-a25a-2d4b68c45164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"
Oct 30 13:00:00.443664 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.443453    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gllsr\" (UniqueName: \"kubernetes.io/projected/e2e00d30-b982-4c55-b2a5-984eb00676ef-kube-api-access-gllsr\") pod \"maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf\" (UID: \"e2e00d30-b982-4c55-b2a5-984eb00676ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"
Oct 30 13:00:00.443664 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.443533    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kfbr\" (UniqueName: \"kubernetes.io/projected/b6bb0c30-e293-4b79-9e36-d37599110835-kube-api-access-8kfbr\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95\" (UID: \"b6bb0c30-e293-4b79-9e36-d37599110835\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"
Oct 30 13:00:00.443664 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.443590    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa90322-42c2-45b4-a25a-2d4b68c45164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w\" (UID: \"0fa90322-42c2-45b4-a25a-2d4b68c45164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"
Oct 30 13:00:00.443864 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.443699    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71007e75-4683-4704-bf4e-11cd7669f1ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz\" (UID: \"71007e75-4683-4704-bf4e-11cd7669f1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"
Oct 30 13:00:00.443864 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.443721    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bb0c30-e293-4b79-9e36-d37599110835-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95\" (UID: \"b6bb0c30-e293-4b79-9e36-d37599110835\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"
Oct 30 13:00:00.443864 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.443804    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e00d30-b982-4c55-b2a5-984eb00676ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf\" (UID: \"e2e00d30-b982-4c55-b2a5-984eb00676ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"
Oct 30 13:00:00.443864 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.443853    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjpbq\" (UniqueName: \"kubernetes.io/projected/2f81fbc0-f46e-4730-8e0d-d44a67f089c8-kube-api-access-xjpbq\") pod \"maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8\" (UID: \"2f81fbc0-f46e-4730-8e0d-d44a67f089c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"
Oct 30 13:00:00.444087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.443914    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f81fbc0-f46e-4730-8e0d-d44a67f089c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8\" (UID: \"2f81fbc0-f46e-4730-8e0d-d44a67f089c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"
Oct 30 13:00:00.444087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.443965    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xtzn\" (UniqueName: \"kubernetes.io/projected/71007e75-4683-4704-bf4e-11cd7669f1ba-kube-api-access-5xtzn\") pod \"maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz\" (UID: \"71007e75-4683-4704-bf4e-11cd7669f1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"
Oct 30 13:00:00.444627 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.444600    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71007e75-4683-4704-bf4e-11cd7669f1ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz\" (UID: \"71007e75-4683-4704-bf4e-11cd7669f1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"
Oct 30 13:00:00.444627 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.444611    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa90322-42c2-45b4-a25a-2d4b68c45164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w\" (UID: \"0fa90322-42c2-45b4-a25a-2d4b68c45164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"
Oct 30 13:00:00.444941 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.444920    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bb0c30-e293-4b79-9e36-d37599110835-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95\" (UID: \"b6bb0c30-e293-4b79-9e36-d37599110835\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"
Oct 30 13:00:00.445178 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.445159    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e00d30-b982-4c55-b2a5-984eb00676ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf\" (UID: \"e2e00d30-b982-4c55-b2a5-984eb00676ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"
Oct 30 13:00:00.445223 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.445199    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f81fbc0-f46e-4730-8e0d-d44a67f089c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8\" (UID: \"2f81fbc0-f46e-4730-8e0d-d44a67f089c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"
Oct 30 13:00:00.466753 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.466728    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gllsr\" (UniqueName: \"kubernetes.io/projected/e2e00d30-b982-4c55-b2a5-984eb00676ef-kube-api-access-gllsr\") pod \"maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf\" (UID: \"e2e00d30-b982-4c55-b2a5-984eb00676ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"
Oct 30 13:00:00.467620 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.467600    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xtzn\" (UniqueName: \"kubernetes.io/projected/71007e75-4683-4704-bf4e-11cd7669f1ba-kube-api-access-5xtzn\") pod \"maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz\" (UID: \"71007e75-4683-4704-bf4e-11cd7669f1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"
Oct 30 13:00:00.468796 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.468774    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjpbq\" (UniqueName: \"kubernetes.io/projected/2f81fbc0-f46e-4730-8e0d-d44a67f089c8-kube-api-access-xjpbq\") pod \"maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8\" (UID: \"2f81fbc0-f46e-4730-8e0d-d44a67f089c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"
Oct 30 13:00:00.478433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.478385    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv6pt\" (UniqueName: \"kubernetes.io/projected/0fa90322-42c2-45b4-a25a-2d4b68c45164-kube-api-access-bv6pt\") pod \"maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w\" (UID: \"0fa90322-42c2-45b4-a25a-2d4b68c45164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"
Oct 30 13:00:00.478846 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.478824    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kfbr\" (UniqueName: \"kubernetes.io/projected/b6bb0c30-e293-4b79-9e36-d37599110835-kube-api-access-8kfbr\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95\" (UID: \"b6bb0c30-e293-4b79-9e36-d37599110835\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"
Oct 30 13:00:00.493551 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.493531    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"
Oct 30 13:00:00.508203 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.508178    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"
Oct 30 13:00:00.532955 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.532680    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"
Oct 30 13:00:00.563852 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.562230    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"
Oct 30 13:00:00.574345 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.574266    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"
Oct 30 13:00:00.775784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.775757    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"
Oct 30 13:00:00.939731 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.939119    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"]
Oct 30 13:00:00.943576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.942905    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"]
Oct 30 13:00:00.945085 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:00:00.945044    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e00d30_b982_4c55_b2a5_984eb00676ef.slice/crio-6b5faf9a3646d2ddd86f1393b89fb0e989724c81d02b482b0363faeee7f00086 WatchSource:0}: Error finding container 6b5faf9a3646d2ddd86f1393b89fb0e989724c81d02b482b0363faeee7f00086: Status 404 returned error can't find the container with id 6b5faf9a3646d2ddd86f1393b89fb0e989724c81d02b482b0363faeee7f00086
Oct 30 13:00:00.980605 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:00:00.980579    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa90322_42c2_45b4_a25a_2d4b68c45164.slice/crio-ecda07bf8ea8aa27e28a3c1b2ec23b176c506bccf51293077ee711bcc8972ed2 WatchSource:0}: Error finding container ecda07bf8ea8aa27e28a3c1b2ec23b176c506bccf51293077ee711bcc8972ed2: Status 404 returned error can't find the container with id ecda07bf8ea8aa27e28a3c1b2ec23b176c506bccf51293077ee711bcc8972ed2
Oct 30 13:00:00.983088 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:00.983002    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"]
Oct 30 13:00:01.114622 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:01.114596    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"]
Oct 30 13:00:01.188509 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:00:01.188437    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bb0c30_e293_4b79_9e36_d37599110835.slice/crio-734a00777edc586d1f0d2831ba64e2a05c6d703ab69736057fc4cc73fbdd189e WatchSource:0}: Error finding container 734a00777edc586d1f0d2831ba64e2a05c6d703ab69736057fc4cc73fbdd189e: Status 404 returned error can't find the container with id 734a00777edc586d1f0d2831ba64e2a05c6d703ab69736057fc4cc73fbdd189e
Oct 30 13:00:01.218839 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:01.218816    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"]
Oct 30 13:00:01.224920 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:01.223875    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"]
Oct 30 13:00:01.273575 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:01.273547    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz" event={"ID":"71007e75-4683-4704-bf4e-11cd7669f1ba","Type":"ContainerStarted","Data":"39c7105f5fb6595755b41b9d63e4e8e68ffc4a1bb8780c344b7ca97d47668692"}
Oct 30 13:00:01.275109 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:01.275085    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf" event={"ID":"e2e00d30-b982-4c55-b2a5-984eb00676ef","Type":"ContainerStarted","Data":"6b5faf9a3646d2ddd86f1393b89fb0e989724c81d02b482b0363faeee7f00086"}
Oct 30 13:00:01.276424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:01.276366    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95" event={"ID":"b6bb0c30-e293-4b79-9e36-d37599110835","Type":"ContainerStarted","Data":"734a00777edc586d1f0d2831ba64e2a05c6d703ab69736057fc4cc73fbdd189e"}
Oct 30 13:00:01.277671 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:01.277651    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8" event={"ID":"2f81fbc0-f46e-4730-8e0d-d44a67f089c8","Type":"ContainerStarted","Data":"d367c623d83c0f15381cde774308df587058821519db1c292570cc30ac8d699a"}
Oct 30 13:00:01.278935 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:01.278915    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27" event={"ID":"84fb8f80-260a-41ee-9df6-3a7f5a21a9df","Type":"ContainerStarted","Data":"adb7490dc70dd48d20b6ba7d3c459b400233149636874e8a3ebb9518882746aa"}
Oct 30 13:00:01.280195 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:01.280175    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w" event={"ID":"0fa90322-42c2-45b4-a25a-2d4b68c45164","Type":"ContainerStarted","Data":"ecda07bf8ea8aa27e28a3c1b2ec23b176c506bccf51293077ee711bcc8972ed2"}
Oct 30 13:00:02.292778 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.292742    5837 generic.go:356] "Generic (PLEG): container finished" podID="0fa90322-42c2-45b4-a25a-2d4b68c45164" containerID="512a3019e0d23c1cd187ec99f32afbdd533ef0d70fa8c29c3acdf9a18ef3a385" exitCode=60
Oct 30 13:00:02.293171 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.292926    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w" event={"ID":"0fa90322-42c2-45b4-a25a-2d4b68c45164","Type":"ContainerDied","Data":"512a3019e0d23c1cd187ec99f32afbdd533ef0d70fa8c29c3acdf9a18ef3a385"}
Oct 30 13:00:02.294158 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.293331    5837 scope.go:122] "RemoveContainer" containerID="512a3019e0d23c1cd187ec99f32afbdd533ef0d70fa8c29c3acdf9a18ef3a385"
Oct 30 13:00:02.295676 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.295654    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz" event={"ID":"71007e75-4683-4704-bf4e-11cd7669f1ba","Type":"ContainerStarted","Data":"58034f0a486223c97938ad6a3cd734942f650a0d5107972222566768df51239a"}
Oct 30 13:00:02.298053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.298029    5837 generic.go:356] "Generic (PLEG): container finished" podID="e2e00d30-b982-4c55-b2a5-984eb00676ef" containerID="9f0c81c9ba1b347aa91af4aa9f4554c87f3d1c63dbcb5a95bac17ac317e1aa68" exitCode=60
Oct 30 13:00:02.298146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.298121    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf" event={"ID":"e2e00d30-b982-4c55-b2a5-984eb00676ef","Type":"ContainerDied","Data":"9f0c81c9ba1b347aa91af4aa9f4554c87f3d1c63dbcb5a95bac17ac317e1aa68"}
Oct 30 13:00:02.298497 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.298477    5837 scope.go:122] "RemoveContainer" containerID="9f0c81c9ba1b347aa91af4aa9f4554c87f3d1c63dbcb5a95bac17ac317e1aa68"
Oct 30 13:00:02.305712 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.304834    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95" event={"ID":"b6bb0c30-e293-4b79-9e36-d37599110835","Type":"ContainerStarted","Data":"ad4312e35fb4b3c13c7ffec3c6ef8f76b733aa67921234036db72741c12e86ed"}
Oct 30 13:00:02.305712 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.305090    5837 scope.go:122] "RemoveContainer" containerID="ad4312e35fb4b3c13c7ffec3c6ef8f76b733aa67921234036db72741c12e86ed"
Oct 30 13:00:02.306699 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.306677    5837 generic.go:356] "Generic (PLEG): container finished" podID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8" containerID="8556835097ea5ed70c751eab71f6ee29bd0061ce38dd642fbb17ed893ca86db5" exitCode=60
Oct 30 13:00:02.306798 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.306747    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8" event={"ID":"2f81fbc0-f46e-4730-8e0d-d44a67f089c8","Type":"ContainerDied","Data":"8556835097ea5ed70c751eab71f6ee29bd0061ce38dd642fbb17ed893ca86db5"}
Oct 30 13:00:02.306991 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.306972    5837 scope.go:122] "RemoveContainer" containerID="8556835097ea5ed70c751eab71f6ee29bd0061ce38dd642fbb17ed893ca86db5"
Oct 30 13:00:02.308976 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.308821    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27" event={"ID":"84fb8f80-260a-41ee-9df6-3a7f5a21a9df","Type":"ContainerStarted","Data":"bb7530c4008d848552e9b173dc21067e2e817d3af14b7e8545900edcf45bc70f"}
Oct 30 13:00:02.309185 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.309078    5837 scope.go:122] "RemoveContainer" containerID="bb7530c4008d848552e9b173dc21067e2e817d3af14b7e8545900edcf45bc70f"
Oct 30 13:00:02.527292 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.527239    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz" podStartSLOduration=2.52722463 podStartE2EDuration="2.52722463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:02.523349423 +0000 UTC m=+1391.047633644" watchObservedRunningTime="2026-10-30 13:00:02.52722463 +0000 UTC m=+1391.051508848"
Oct 30 13:00:02.906846 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.906785    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"]
Oct 30 13:00:02.912749 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.912727    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"]
Oct 30 13:00:02.913014 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.912987    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"
Oct 30 13:00:02.928466 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:02.928444    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:00:03.004841 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.004807    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739294c5-2be4-4958-891a-bf63f2081bef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc\" (UID: \"739294c5-2be4-4958-891a-bf63f2081bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"
Oct 30 13:00:03.005007 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.004989    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvhkg\" (UniqueName: \"kubernetes.io/projected/739294c5-2be4-4958-891a-bf63f2081bef-kube-api-access-hvhkg\") pod \"maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc\" (UID: \"739294c5-2be4-4958-891a-bf63f2081bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"
Oct 30 13:00:03.005075 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.005063    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739294c5-2be4-4958-891a-bf63f2081bef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc\" (UID: \"739294c5-2be4-4958-891a-bf63f2081bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"
Oct 30 13:00:03.106462 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.106432    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvhkg\" (UniqueName: \"kubernetes.io/projected/739294c5-2be4-4958-891a-bf63f2081bef-kube-api-access-hvhkg\") pod \"maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc\" (UID: \"739294c5-2be4-4958-891a-bf63f2081bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"
Oct 30 13:00:03.106596 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.106510    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739294c5-2be4-4958-891a-bf63f2081bef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc\" (UID: \"739294c5-2be4-4958-891a-bf63f2081bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"
Oct 30 13:00:03.106647 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.106605    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739294c5-2be4-4958-891a-bf63f2081bef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc\" (UID: \"739294c5-2be4-4958-891a-bf63f2081bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"
Oct 30 13:00:03.107216 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.107138    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739294c5-2be4-4958-891a-bf63f2081bef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc\" (UID: \"739294c5-2be4-4958-891a-bf63f2081bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"
Oct 30 13:00:03.108908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.108887    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739294c5-2be4-4958-891a-bf63f2081bef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc\" (UID: \"739294c5-2be4-4958-891a-bf63f2081bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"
Oct 30 13:00:03.125153 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.125126    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvhkg\" (UniqueName: \"kubernetes.io/projected/739294c5-2be4-4958-891a-bf63f2081bef-kube-api-access-hvhkg\") pod \"maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc\" (UID: \"739294c5-2be4-4958-891a-bf63f2081bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"
Oct 30 13:00:03.229145 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.229081    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"
Oct 30 13:00:03.325775 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.325541    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w" event={"ID":"0fa90322-42c2-45b4-a25a-2d4b68c45164","Type":"ContainerStarted","Data":"3560efcdbf486bbd67393e6a24ddf603284126b8e3d7322c74a8081d7f5977b2"}
Oct 30 13:00:03.329237 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.327826    5837 generic.go:356] "Generic (PLEG): container finished" podID="71007e75-4683-4704-bf4e-11cd7669f1ba" containerID="58034f0a486223c97938ad6a3cd734942f650a0d5107972222566768df51239a" exitCode=60
Oct 30 13:00:03.329237 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.327890    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz" event={"ID":"71007e75-4683-4704-bf4e-11cd7669f1ba","Type":"ContainerDied","Data":"58034f0a486223c97938ad6a3cd734942f650a0d5107972222566768df51239a"}
Oct 30 13:00:03.329237 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.328157    5837 scope.go:122] "RemoveContainer" containerID="58034f0a486223c97938ad6a3cd734942f650a0d5107972222566768df51239a"
Oct 30 13:00:03.331351 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.331280    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf" event={"ID":"e2e00d30-b982-4c55-b2a5-984eb00676ef","Type":"ContainerStarted","Data":"06d88bc24d4a35b18f0c362bf1e2d0cf5acc813f47518df3a5b36b5bf9cd4f63"}
Oct 30 13:00:03.335640 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.334665    5837 generic.go:356] "Generic (PLEG): container finished" podID="b6bb0c30-e293-4b79-9e36-d37599110835" containerID="ad4312e35fb4b3c13c7ffec3c6ef8f76b733aa67921234036db72741c12e86ed" exitCode=60
Oct 30 13:00:03.335640 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.334740    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95" event={"ID":"b6bb0c30-e293-4b79-9e36-d37599110835","Type":"ContainerDied","Data":"ad4312e35fb4b3c13c7ffec3c6ef8f76b733aa67921234036db72741c12e86ed"}
Oct 30 13:00:03.335640 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.334769    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95" event={"ID":"b6bb0c30-e293-4b79-9e36-d37599110835","Type":"ContainerStarted","Data":"773805f81f26fa3ddae6619c548bcd9f091211dcd46acd6aa77b2c519415b3b2"}
Oct 30 13:00:03.337364 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.337326    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8" event={"ID":"2f81fbc0-f46e-4730-8e0d-d44a67f089c8","Type":"ContainerStarted","Data":"d052a530f9cc91b132bfefd7002a9b6bcee8d0561d4e8f8ada919a05c0152a8e"}
Oct 30 13:00:03.339282 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.339257    5837 generic.go:356] "Generic (PLEG): container finished" podID="84fb8f80-260a-41ee-9df6-3a7f5a21a9df" containerID="bb7530c4008d848552e9b173dc21067e2e817d3af14b7e8545900edcf45bc70f" exitCode=60
Oct 30 13:00:03.339453 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.339348    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27" event={"ID":"84fb8f80-260a-41ee-9df6-3a7f5a21a9df","Type":"ContainerDied","Data":"bb7530c4008d848552e9b173dc21067e2e817d3af14b7e8545900edcf45bc70f"}
Oct 30 13:00:03.339453 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.339425    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27" event={"ID":"84fb8f80-260a-41ee-9df6-3a7f5a21a9df","Type":"ContainerStarted","Data":"00209ead39aa0717387538120b04b207f222a2aa773fafc28c20e0583c56705d"}
Oct 30 13:00:03.368752 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.367764    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w" podStartSLOduration=3.367745049 podStartE2EDuration="3.367745049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:03.364569609 +0000 UTC m=+1391.888853830" watchObservedRunningTime="2026-10-30 13:00:03.367745049 +0000 UTC m=+1391.892029270"
Oct 30 13:00:03.404270 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.404232    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf" podStartSLOduration=3.404219035 podStartE2EDuration="3.404219035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:03.40111102 +0000 UTC m=+1391.925395267" watchObservedRunningTime="2026-10-30 13:00:03.404219035 +0000 UTC m=+1391.928503254"
Oct 30 13:00:03.443340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.443295    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27" podStartSLOduration=3.443282121 podStartE2EDuration="3.443282121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:03.439169084 +0000 UTC m=+1391.963453304" watchObservedRunningTime="2026-10-30 13:00:03.443282121 +0000 UTC m=+1391.967566339"
Oct 30 13:00:03.520096 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.520042    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95" podStartSLOduration=3.520024121 podStartE2EDuration="3.520024121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:03.513728113 +0000 UTC m=+1392.038012361" watchObservedRunningTime="2026-10-30 13:00:03.520024121 +0000 UTC m=+1392.044308346"
Oct 30 13:00:03.540010 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.539985    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"]
Oct 30 13:00:03.552699 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:03.552661    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8" podStartSLOduration=3.552649123 podStartE2EDuration="3.552649123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:03.551331338 +0000 UTC m=+1392.075615576" watchObservedRunningTime="2026-10-30 13:00:03.552649123 +0000 UTC m=+1392.076933341"
Oct 30 13:00:03.571469 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:00:03.571443    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739294c5_2be4_4958_891a_bf63f2081bef.slice/crio-b738b70f9e934e5d45279ec8cf01b2a138159ddbf18d8e74ff1e2f8d83c01b4f WatchSource:0}: Error finding container b738b70f9e934e5d45279ec8cf01b2a138159ddbf18d8e74ff1e2f8d83c01b4f: Status 404 returned error can't find the container with id b738b70f9e934e5d45279ec8cf01b2a138159ddbf18d8e74ff1e2f8d83c01b4f
Oct 30 13:00:04.353588 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.353553    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc" event={"ID":"739294c5-2be4-4958-891a-bf63f2081bef","Type":"ContainerStarted","Data":"778afdf036135bfd2d3916c84d72553328e1bf4c4b86a16e045b44fa15d6ad30"}
Oct 30 13:00:04.354043 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.353614    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc" event={"ID":"739294c5-2be4-4958-891a-bf63f2081bef","Type":"ContainerStarted","Data":"b738b70f9e934e5d45279ec8cf01b2a138159ddbf18d8e74ff1e2f8d83c01b4f"}
Oct 30 13:00:04.355677 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.355649    5837 generic.go:356] "Generic (PLEG): container finished" podID="0fa90322-42c2-45b4-a25a-2d4b68c45164" containerID="3560efcdbf486bbd67393e6a24ddf603284126b8e3d7322c74a8081d7f5977b2" exitCode=60
Oct 30 13:00:04.355832 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.355740    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w" event={"ID":"0fa90322-42c2-45b4-a25a-2d4b68c45164","Type":"ContainerDied","Data":"3560efcdbf486bbd67393e6a24ddf603284126b8e3d7322c74a8081d7f5977b2"}
Oct 30 13:00:04.355832 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.355798    5837 scope.go:122] "RemoveContainer" containerID="512a3019e0d23c1cd187ec99f32afbdd533ef0d70fa8c29c3acdf9a18ef3a385"
Oct 30 13:00:04.356024 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.356007    5837 scope.go:122] "RemoveContainer" containerID="3560efcdbf486bbd67393e6a24ddf603284126b8e3d7322c74a8081d7f5977b2"
Oct 30 13:00:04.356382 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:04.356345    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w_odh-ai-gateway-infra(0fa90322-42c2-45b4-a25a-2d4b68c45164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w" podUID="0fa90322-42c2-45b4-a25a-2d4b68c45164"
Oct 30 13:00:04.358256 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.358236    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz" event={"ID":"71007e75-4683-4704-bf4e-11cd7669f1ba","Type":"ContainerStarted","Data":"7ef3737ed1a69dff58e197c471cd50c3ce5de183a091a08e0214c282274c3506"}
Oct 30 13:00:04.361048 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.361024    5837 generic.go:356] "Generic (PLEG): container finished" podID="e2e00d30-b982-4c55-b2a5-984eb00676ef" containerID="06d88bc24d4a35b18f0c362bf1e2d0cf5acc813f47518df3a5b36b5bf9cd4f63" exitCode=60
Oct 30 13:00:04.361167 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.361117    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf" event={"ID":"e2e00d30-b982-4c55-b2a5-984eb00676ef","Type":"ContainerDied","Data":"06d88bc24d4a35b18f0c362bf1e2d0cf5acc813f47518df3a5b36b5bf9cd4f63"}
Oct 30 13:00:04.361432 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.361369    5837 scope.go:122] "RemoveContainer" containerID="06d88bc24d4a35b18f0c362bf1e2d0cf5acc813f47518df3a5b36b5bf9cd4f63"
Oct 30 13:00:04.361770 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:04.361726    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf_odh-ai-gateway-infra(e2e00d30-b982-4c55-b2a5-984eb00676ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf" podUID="e2e00d30-b982-4c55-b2a5-984eb00676ef"
Oct 30 13:00:04.363301 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.363266    5837 generic.go:356] "Generic (PLEG): container finished" podID="b6bb0c30-e293-4b79-9e36-d37599110835" containerID="773805f81f26fa3ddae6619c548bcd9f091211dcd46acd6aa77b2c519415b3b2" exitCode=60
Oct 30 13:00:04.363432 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.363353    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95" event={"ID":"b6bb0c30-e293-4b79-9e36-d37599110835","Type":"ContainerDied","Data":"773805f81f26fa3ddae6619c548bcd9f091211dcd46acd6aa77b2c519415b3b2"}
Oct 30 13:00:04.363633 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.363616    5837 scope.go:122] "RemoveContainer" containerID="773805f81f26fa3ddae6619c548bcd9f091211dcd46acd6aa77b2c519415b3b2"
Oct 30 13:00:04.363950 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:04.363929    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95_odh-ai-gateway-infra(b6bb0c30-e293-4b79-9e36-d37599110835)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95" podUID="b6bb0c30-e293-4b79-9e36-d37599110835"
Oct 30 13:00:04.365638 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.365618    5837 generic.go:356] "Generic (PLEG): container finished" podID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8" containerID="d052a530f9cc91b132bfefd7002a9b6bcee8d0561d4e8f8ada919a05c0152a8e" exitCode=60
Oct 30 13:00:04.365756 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.365709    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8" event={"ID":"2f81fbc0-f46e-4730-8e0d-d44a67f089c8","Type":"ContainerDied","Data":"d052a530f9cc91b132bfefd7002a9b6bcee8d0561d4e8f8ada919a05c0152a8e"}
Oct 30 13:00:04.365904 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.365890    5837 scope.go:122] "RemoveContainer" containerID="d052a530f9cc91b132bfefd7002a9b6bcee8d0561d4e8f8ada919a05c0152a8e"
Oct 30 13:00:04.366166 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:04.366148    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8_odh-ai-gateway-infra(2f81fbc0-f46e-4730-8e0d-d44a67f089c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8" podUID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8"
Oct 30 13:00:04.367785 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.367765    5837 generic.go:356] "Generic (PLEG): container finished" podID="84fb8f80-260a-41ee-9df6-3a7f5a21a9df" containerID="00209ead39aa0717387538120b04b207f222a2aa773fafc28c20e0583c56705d" exitCode=60
Oct 30 13:00:04.367895 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.367787    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27" event={"ID":"84fb8f80-260a-41ee-9df6-3a7f5a21a9df","Type":"ContainerDied","Data":"00209ead39aa0717387538120b04b207f222a2aa773fafc28c20e0583c56705d"}
Oct 30 13:00:04.368021 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.368007    5837 scope.go:122] "RemoveContainer" containerID="00209ead39aa0717387538120b04b207f222a2aa773fafc28c20e0583c56705d"
Oct 30 13:00:04.368248 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:04.368230    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27_odh-ai-gateway-infra(84fb8f80-260a-41ee-9df6-3a7f5a21a9df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27" podUID="84fb8f80-260a-41ee-9df6-3a7f5a21a9df"
Oct 30 13:00:04.371961 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.371945    5837 scope.go:122] "RemoveContainer" containerID="9f0c81c9ba1b347aa91af4aa9f4554c87f3d1c63dbcb5a95bac17ac317e1aa68"
Oct 30 13:00:04.385930 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.385906    5837 scope.go:122] "RemoveContainer" containerID="ad4312e35fb4b3c13c7ffec3c6ef8f76b733aa67921234036db72741c12e86ed"
Oct 30 13:00:04.397227 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.397152    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc" podStartSLOduration=2.397135993 podStartE2EDuration="2.397135993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:04.395253348 +0000 UTC m=+1392.919537566" watchObservedRunningTime="2026-10-30 13:00:04.397135993 +0000 UTC m=+1392.921420212"
Oct 30 13:00:04.401437 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.401381    5837 scope.go:122] "RemoveContainer" containerID="8556835097ea5ed70c751eab71f6ee29bd0061ce38dd642fbb17ed893ca86db5"
Oct 30 13:00:04.415047 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:04.415026    5837 scope.go:122] "RemoveContainer" containerID="bb7530c4008d848552e9b173dc21067e2e817d3af14b7e8545900edcf45bc70f"
Oct 30 13:00:05.380753 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:05.380721    5837 scope.go:122] "RemoveContainer" containerID="06d88bc24d4a35b18f0c362bf1e2d0cf5acc813f47518df3a5b36b5bf9cd4f63"
Oct 30 13:00:05.381239 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:05.381144    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf_odh-ai-gateway-infra(e2e00d30-b982-4c55-b2a5-984eb00676ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf" podUID="e2e00d30-b982-4c55-b2a5-984eb00676ef"
Oct 30 13:00:05.382695 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:05.382671    5837 scope.go:122] "RemoveContainer" containerID="773805f81f26fa3ddae6619c548bcd9f091211dcd46acd6aa77b2c519415b3b2"
Oct 30 13:00:05.383031 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:05.383009    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95_odh-ai-gateway-infra(b6bb0c30-e293-4b79-9e36-d37599110835)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95" podUID="b6bb0c30-e293-4b79-9e36-d37599110835"
Oct 30 13:00:05.384464 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:05.384429    5837 scope.go:122] "RemoveContainer" containerID="d052a530f9cc91b132bfefd7002a9b6bcee8d0561d4e8f8ada919a05c0152a8e"
Oct 30 13:00:05.384755 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:05.384735    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8_odh-ai-gateway-infra(2f81fbc0-f46e-4730-8e0d-d44a67f089c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8" podUID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8"
Oct 30 13:00:05.386085 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:05.386062    5837 scope.go:122] "RemoveContainer" containerID="00209ead39aa0717387538120b04b207f222a2aa773fafc28c20e0583c56705d"
Oct 30 13:00:05.386447 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:05.386423    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27_odh-ai-gateway-infra(84fb8f80-260a-41ee-9df6-3a7f5a21a9df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27" podUID="84fb8f80-260a-41ee-9df6-3a7f5a21a9df"
Oct 30 13:00:05.387457 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:05.387433    5837 generic.go:356] "Generic (PLEG): container finished" podID="739294c5-2be4-4958-891a-bf63f2081bef" containerID="778afdf036135bfd2d3916c84d72553328e1bf4c4b86a16e045b44fa15d6ad30" exitCode=0
Oct 30 13:00:05.387572 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:05.387482    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc" event={"ID":"739294c5-2be4-4958-891a-bf63f2081bef","Type":"ContainerDied","Data":"778afdf036135bfd2d3916c84d72553328e1bf4c4b86a16e045b44fa15d6ad30"}
Oct 30 13:00:05.389240 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:05.389224    5837 scope.go:122] "RemoveContainer" containerID="3560efcdbf486bbd67393e6a24ddf603284126b8e3d7322c74a8081d7f5977b2"
Oct 30 13:00:05.389564 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:05.389545    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w_odh-ai-gateway-infra(0fa90322-42c2-45b4-a25a-2d4b68c45164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w" podUID="0fa90322-42c2-45b4-a25a-2d4b68c45164"
Oct 30 13:00:05.390773 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:05.390754    5837 generic.go:356] "Generic (PLEG): container finished" podID="71007e75-4683-4704-bf4e-11cd7669f1ba" containerID="7ef3737ed1a69dff58e197c471cd50c3ce5de183a091a08e0214c282274c3506" exitCode=60
Oct 30 13:00:05.390851 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:05.390780    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz" event={"ID":"71007e75-4683-4704-bf4e-11cd7669f1ba","Type":"ContainerDied","Data":"7ef3737ed1a69dff58e197c471cd50c3ce5de183a091a08e0214c282274c3506"}
Oct 30 13:00:05.390851 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:05.390813    5837 scope.go:122] "RemoveContainer" containerID="58034f0a486223c97938ad6a3cd734942f650a0d5107972222566768df51239a"
Oct 30 13:00:05.391035 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:05.391020    5837 scope.go:122] "RemoveContainer" containerID="7ef3737ed1a69dff58e197c471cd50c3ce5de183a091a08e0214c282274c3506"
Oct 30 13:00:05.391284 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:05.391259    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz_odh-ai-gateway-infra(71007e75-4683-4704-bf4e-11cd7669f1ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz" podUID="71007e75-4683-4704-bf4e-11cd7669f1ba"
Oct 30 13:00:06.400875 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:06.400848    5837 scope.go:122] "RemoveContainer" containerID="7ef3737ed1a69dff58e197c471cd50c3ce5de183a091a08e0214c282274c3506"
Oct 30 13:00:06.401589 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:06.401261    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz_odh-ai-gateway-infra(71007e75-4683-4704-bf4e-11cd7669f1ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz" podUID="71007e75-4683-4704-bf4e-11cd7669f1ba"
Oct 30 13:00:06.653644 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:06.653580    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"
Oct 30 13:00:06.768373 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:06.768342    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/739294c5-2be4-4958-891a-bf63f2081bef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739294c5-2be4-4958-891a-bf63f2081bef-maas-api-xks-ca\") pod \"739294c5-2be4-4958-891a-bf63f2081bef\" (UID: \"739294c5-2be4-4958-891a-bf63f2081bef\") "
Oct 30 13:00:06.768632 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:06.768600    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739294c5-2be4-4958-891a-bf63f2081bef-kube-api-access-hvhkg\" (UniqueName: \"kubernetes.io/projected/739294c5-2be4-4958-891a-bf63f2081bef-kube-api-access-hvhkg\") pod \"739294c5-2be4-4958-891a-bf63f2081bef\" (UID: \"739294c5-2be4-4958-891a-bf63f2081bef\") "
Oct 30 13:00:06.768760 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:06.768692    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739294c5-2be4-4958-891a-bf63f2081bef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739294c5-2be4-4958-891a-bf63f2081bef-maas-api-openshift-ca\") pod \"739294c5-2be4-4958-891a-bf63f2081bef\" (UID: \"739294c5-2be4-4958-891a-bf63f2081bef\") "
Oct 30 13:00:06.769089 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:06.769065    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739294c5-2be4-4958-891a-bf63f2081bef-maas-api-openshift-ca" pod "739294c5-2be4-4958-891a-bf63f2081bef" (UID: "739294c5-2be4-4958-891a-bf63f2081bef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:06.769321 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:06.769272    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739294c5-2be4-4958-891a-bf63f2081bef-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:06.770757 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:06.770732    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/739294c5-2be4-4958-891a-bf63f2081bef-maas-api-xks-ca" pod "739294c5-2be4-4958-891a-bf63f2081bef" (UID: "739294c5-2be4-4958-891a-bf63f2081bef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:00:06.770921 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:06.770897    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739294c5-2be4-4958-891a-bf63f2081bef-kube-api-access-hvhkg" pod "739294c5-2be4-4958-891a-bf63f2081bef" (UID: "739294c5-2be4-4958-891a-bf63f2081bef"). InnerVolumeSpecName "kube-api-access-hvhkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:06.871153 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:06.871128    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvhkg\" (UniqueName: \"kubernetes.io/projected/739294c5-2be4-4958-891a-bf63f2081bef-kube-api-access-hvhkg\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:06.871153 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:06.871153    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739294c5-2be4-4958-891a-bf63f2081bef-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:07.027411 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:07.027359    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"]
Oct 30 13:00:07.031875 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:07.031852    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-604a4dfa-rf6js"]
Oct 30 13:00:07.426259 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:07.424247    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc" event={"ID":"739294c5-2be4-4958-891a-bf63f2081bef","Type":"ContainerDied","Data":"b738b70f9e934e5d45279ec8cf01b2a138159ddbf18d8e74ff1e2f8d83c01b4f"}
Oct 30 13:00:07.426861 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:07.426839    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b738b70f9e934e5d45279ec8cf01b2a138159ddbf18d8e74ff1e2f8d83c01b4f"
Oct 30 13:00:07.428670 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:07.427476    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"
Oct 30 13:00:07.612365 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:07.612332    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b93a9a05-8a65-4ca1-a749-cf7e7c476240" path="/var/lib/kubelet/pods/b93a9a05-8a65-4ca1-a749-cf7e7c476240/volumes"
Oct 30 13:00:09.271418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:09.264581    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"
Oct 30 13:00:09.278420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:09.272688    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"
Oct 30 13:00:11.296415 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:11.285353    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"
Oct 30 13:00:11.313439 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:11.305498    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"
Oct 30 13:00:11.358942 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:11.358900    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:11.360170 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:11.360141    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:11.783324 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:11.770729    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:12.363631 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:12.363586    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:00:12.371213 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:12.369881    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:00:12.408792 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:12.407627    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:12.408792 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:12.408117    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 13:00:13.070421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:13.070336    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:13.086431 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:13.070923    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 13:00:13.372487 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:13.369547    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:13.418435 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:13.414863    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 13:00:14.074555 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:14.074486    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:00:14.403318 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:14.403213    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:00:14.433416 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:14.429701    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:14.467978 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:14.467849    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:14.833423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:14.823266    5837 scope.go:122] "RemoveContainer" containerID="3560efcdbf486bbd67393e6a24ddf603284126b8e3d7322c74a8081d7f5977b2"
Oct 30 13:00:15.096934 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:15.096890    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:15.505423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:15.472141    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 13:00:15.537170 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:15.519376    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:00:15.608502 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:15.608470    5837 scope.go:122] "RemoveContainer" containerID="00209ead39aa0717387538120b04b207f222a2aa773fafc28c20e0583c56705d"
Oct 30 13:00:15.774501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:15.756530    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:00:15.774501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:15.756592    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" podUID="25644850-37a6-4820-8ed2-4f00fedd8274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:16.144477 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:16.142857    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:00:16.412646 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:16.405496    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:16.562433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:16.560218    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 13:00:16.562997 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:16.562951    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:16.603884 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:16.603846    5837 scope.go:122] "RemoveContainer" containerID="d052a530f9cc91b132bfefd7002a9b6bcee8d0561d4e8f8ada919a05c0152a8e"
Oct 30 13:00:16.613427 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:16.609016    5837 scope.go:122] "RemoveContainer" containerID="7ef3737ed1a69dff58e197c471cd50c3ce5de183a091a08e0214c282274c3506"
Oct 30 13:00:16.653933 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:16.653900    5837 scope.go:122] "RemoveContainer" containerID="773805f81f26fa3ddae6619c548bcd9f091211dcd46acd6aa77b2c519415b3b2"
Oct 30 13:00:16.698437 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:16.698380    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"]
Oct 30 13:00:16.705863 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:16.705827    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5" podUID="d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" containerName="istio-proxy" containerID="cri-o://4e4f77c9e3572a67ac3665d35bc445bfb6f4e71d412e66fe591ff12cc0ea0684" gracePeriod=30
Oct 30 13:00:17.299736 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:17.299614    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:00:17.300034 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:17.299982    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:17.389034 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:17.388992    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:00:17.389295 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:17.389240    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4" podUID="f855b07d-e71c-491e-b430-0d33bdd77a5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:17.406566 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:17.406528    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:17.624482 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:17.573292    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:17.624482 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:17.573713    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 13:00:17.634884 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:17.631987    5837 scope.go:122] "RemoveContainer" containerID="06d88bc24d4a35b18f0c362bf1e2d0cf5acc813f47518df3a5b36b5bf9cd4f63"
Oct 30 13:00:18.528452 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:18.422142    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:18.573714 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:18.573653    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:18.641085 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:18.612357    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:19.426427 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:19.414085    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27" event={"ID":"84fb8f80-260a-41ee-9df6-3a7f5a21a9df","Type":"ContainerStarted","Data":"74d4cc7509b72ac885835487d4de13fd8b464d4a147d657907a85b3d67acab70"}
Oct 30 13:00:19.472349 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:19.438592    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa90322_42c2_45b4_a25a_2d4b68c45164.slice/crio-conmon-dc73fb04782cc8a3922825a7188da7cd7b260f9b04c13967ec5ef377f722c654.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:00:19.493319 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:19.493279    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:00:19.566420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:19.540594    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:19.570386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:19.570354    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w" event={"ID":"0fa90322-42c2-45b4-a25a-2d4b68c45164","Type":"ContainerStarted","Data":"dc73fb04782cc8a3922825a7188da7cd7b260f9b04c13967ec5ef377f722c654"}
Oct 30 13:00:19.582420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:19.579065    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:00:19.617420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:19.616762    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:19.654301 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:19.654267    5837 scope.go:122] "RemoveContainer" containerID="dc73fb04782cc8a3922825a7188da7cd7b260f9b04c13967ec5ef377f722c654"
Oct 30 13:00:19.655338 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:19.655295    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w_odh-ai-gateway-infra(0fa90322-42c2-45b4-a25a-2d4b68c45164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w" podUID="0fa90322-42c2-45b4-a25a-2d4b68c45164"
Oct 30 13:00:20.104436 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.093582    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"]
Oct 30 13:00:20.118421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.113241    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4beb2220-hv6bl"]
Oct 30 13:00:20.152227 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.148832    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 13:00:20.302760 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.299943    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:00:20.319457 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.310007    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 13:00:20.478222 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.478185    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"]
Oct 30 13:00:20.647239 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.647161    5837 generic.go:356] "Generic (PLEG): container finished" podID="0fa90322-42c2-45b4-a25a-2d4b68c45164" containerID="dc73fb04782cc8a3922825a7188da7cd7b260f9b04c13967ec5ef377f722c654" exitCode=60
Oct 30 13:00:20.647563 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.647228    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w" event={"ID":"0fa90322-42c2-45b4-a25a-2d4b68c45164","Type":"ContainerDied","Data":"dc73fb04782cc8a3922825a7188da7cd7b260f9b04c13967ec5ef377f722c654"}
Oct 30 13:00:20.647764 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.647734    5837 scope.go:122] "RemoveContainer" containerID="3560efcdbf486bbd67393e6a24ddf603284126b8e3d7322c74a8081d7f5977b2"
Oct 30 13:00:20.648585 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.648567    5837 scope.go:122] "RemoveContainer" containerID="dc73fb04782cc8a3922825a7188da7cd7b260f9b04c13967ec5ef377f722c654"
Oct 30 13:00:20.649273 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:20.649248    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w_odh-ai-gateway-infra(0fa90322-42c2-45b4-a25a-2d4b68c45164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w" podUID="0fa90322-42c2-45b4-a25a-2d4b68c45164"
Oct 30 13:00:20.650308 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.650285    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"]
Oct 30 13:00:20.655865 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.655845    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz" event={"ID":"71007e75-4683-4704-bf4e-11cd7669f1ba","Type":"ContainerDied","Data":"4aa8c2d3ed161cc15bad99a46c0c03ee4b04d03410ccb3da5a471c1e006f32ea"}
Oct 30 13:00:20.657247 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.657201    5837 generic.go:356] "Generic (PLEG): container finished" podID="71007e75-4683-4704-bf4e-11cd7669f1ba" containerID="4aa8c2d3ed161cc15bad99a46c0c03ee4b04d03410ccb3da5a471c1e006f32ea" exitCode=60
Oct 30 13:00:20.657812 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.657787    5837 scope.go:122] "RemoveContainer" containerID="4aa8c2d3ed161cc15bad99a46c0c03ee4b04d03410ccb3da5a471c1e006f32ea"
Oct 30 13:00:20.658203 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:20.658175    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz_odh-ai-gateway-infra(71007e75-4683-4704-bf4e-11cd7669f1ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz" podUID="71007e75-4683-4704-bf4e-11cd7669f1ba"
Oct 30 13:00:20.660605 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.660585    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf" event={"ID":"e2e00d30-b982-4c55-b2a5-984eb00676ef","Type":"ContainerStarted","Data":"1d938c94fcdfdc12c9aaff3d0898890222aa7962fa87e03a7c4dc142fcd3bcb5"}
Oct 30 13:00:20.670803 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.667197    5837 generic.go:356] "Generic (PLEG): container finished" podID="b6bb0c30-e293-4b79-9e36-d37599110835" containerID="81b9ce84735ad8138401c2d241090dec95f68ed1aab0eb872400ef258f657e13" exitCode=60
Oct 30 13:00:20.670996 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.670980    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95" event={"ID":"b6bb0c30-e293-4b79-9e36-d37599110835","Type":"ContainerDied","Data":"81b9ce84735ad8138401c2d241090dec95f68ed1aab0eb872400ef258f657e13"}
Oct 30 13:00:20.671507 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.671489    5837 scope.go:122] "RemoveContainer" containerID="81b9ce84735ad8138401c2d241090dec95f68ed1aab0eb872400ef258f657e13"
Oct 30 13:00:20.671997 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:20.671972    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95_odh-ai-gateway-infra(b6bb0c30-e293-4b79-9e36-d37599110835)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95" podUID="b6bb0c30-e293-4b79-9e36-d37599110835"
Oct 30 13:00:20.672685 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.672665    5837 scope.go:122] "RemoveContainer" containerID="7ef3737ed1a69dff58e197c471cd50c3ce5de183a091a08e0214c282274c3506"
Oct 30 13:00:20.677063 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.677042    5837 generic.go:356] "Generic (PLEG): container finished" podID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8" containerID="1d83375388f38e5659278633c8dfb0fe3b08e5a5d0e511338534d1681a8c4cc5" exitCode=60
Oct 30 13:00:20.677280 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.677264    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8" event={"ID":"2f81fbc0-f46e-4730-8e0d-d44a67f089c8","Type":"ContainerDied","Data":"1d83375388f38e5659278633c8dfb0fe3b08e5a5d0e511338534d1681a8c4cc5"}
Oct 30 13:00:20.679026 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.679011    5837 scope.go:122] "RemoveContainer" containerID="1d83375388f38e5659278633c8dfb0fe3b08e5a5d0e511338534d1681a8c4cc5"
Oct 30 13:00:20.680013 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:20.679989    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8_odh-ai-gateway-infra(2f81fbc0-f46e-4730-8e0d-d44a67f089c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8" podUID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8"
Oct 30 13:00:20.681032 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.681009    5837 generic.go:356] "Generic (PLEG): container finished" podID="84fb8f80-260a-41ee-9df6-3a7f5a21a9df" containerID="74d4cc7509b72ac885835487d4de13fd8b464d4a147d657907a85b3d67acab70" exitCode=60
Oct 30 13:00:20.681134 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.681076    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27" event={"ID":"84fb8f80-260a-41ee-9df6-3a7f5a21a9df","Type":"ContainerDied","Data":"74d4cc7509b72ac885835487d4de13fd8b464d4a147d657907a85b3d67acab70"}
Oct 30 13:00:20.730612 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.730568    5837 scope.go:122] "RemoveContainer" containerID="773805f81f26fa3ddae6619c548bcd9f091211dcd46acd6aa77b2c519415b3b2"
Oct 30 13:00:20.826521 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.807434    5837 scope.go:122] "RemoveContainer" containerID="d052a530f9cc91b132bfefd7002a9b6bcee8d0561d4e8f8ada919a05c0152a8e"
Oct 30 13:00:20.873713 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:20.873678    5837 scope.go:122] "RemoveContainer" containerID="00209ead39aa0717387538120b04b207f222a2aa773fafc28c20e0583c56705d"
Oct 30 13:00:21.022291 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.022266    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"
Oct 30 13:00:21.068419 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.054558    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fb8f80-260a-41ee-9df6-3a7f5a21a9df-kube-api-access-6krb4\" (UniqueName: \"kubernetes.io/projected/84fb8f80-260a-41ee-9df6-3a7f5a21a9df-kube-api-access-6krb4\") pod \"84fb8f80-260a-41ee-9df6-3a7f5a21a9df\" (UID: \"84fb8f80-260a-41ee-9df6-3a7f5a21a9df\") "
Oct 30 13:00:21.068419 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.054670    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84fb8f80-260a-41ee-9df6-3a7f5a21a9df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fb8f80-260a-41ee-9df6-3a7f5a21a9df-maas-api-service-ca\") pod \"84fb8f80-260a-41ee-9df6-3a7f5a21a9df\" (UID: \"84fb8f80-260a-41ee-9df6-3a7f5a21a9df\") "
Oct 30 13:00:21.078004 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.077951    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fb8f80-260a-41ee-9df6-3a7f5a21a9df-kube-api-access-6krb4" pod "84fb8f80-260a-41ee-9df6-3a7f5a21a9df" (UID: "84fb8f80-260a-41ee-9df6-3a7f5a21a9df"). InnerVolumeSpecName "kube-api-access-6krb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:21.082692 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.082653    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fb8f80-260a-41ee-9df6-3a7f5a21a9df-maas-api-service-ca" pod "84fb8f80-260a-41ee-9df6-3a7f5a21a9df" (UID: "84fb8f80-260a-41ee-9df6-3a7f5a21a9df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:21.156957 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.156915    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6krb4\" (UniqueName: \"kubernetes.io/projected/84fb8f80-260a-41ee-9df6-3a7f5a21a9df-kube-api-access-6krb4\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:21.156957 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.156954    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fb8f80-260a-41ee-9df6-3a7f5a21a9df-maas-api-service-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:21.624672 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.624622    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7244a952-78d0-4156-ab8b-95bb89a4617b" path="/var/lib/kubelet/pods/7244a952-78d0-4156-ab8b-95bb89a4617b/volumes"
Oct 30 13:00:21.738909 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.738876    5837 generic.go:356] "Generic (PLEG): container finished" podID="e2e00d30-b982-4c55-b2a5-984eb00676ef" containerID="1d938c94fcdfdc12c9aaff3d0898890222aa7962fa87e03a7c4dc142fcd3bcb5" exitCode=60
Oct 30 13:00:21.739387 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.738957    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf" event={"ID":"e2e00d30-b982-4c55-b2a5-984eb00676ef","Type":"ContainerDied","Data":"1d938c94fcdfdc12c9aaff3d0898890222aa7962fa87e03a7c4dc142fcd3bcb5"}
Oct 30 13:00:21.739387 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.739028    5837 scope.go:122] "RemoveContainer" containerID="06d88bc24d4a35b18f0c362bf1e2d0cf5acc813f47518df3a5b36b5bf9cd4f63"
Oct 30 13:00:21.739555 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.739427    5837 scope.go:122] "RemoveContainer" containerID="1d938c94fcdfdc12c9aaff3d0898890222aa7962fa87e03a7c4dc142fcd3bcb5"
Oct 30 13:00:21.739858 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:21.739822    5837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf_odh-ai-gateway-infra(e2e00d30-b982-4c55-b2a5-984eb00676ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf" podUID="e2e00d30-b982-4c55-b2a5-984eb00676ef"
Oct 30 13:00:21.746541 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.746520    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"
Oct 30 13:00:21.746635 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.746544    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27" event={"ID":"84fb8f80-260a-41ee-9df6-3a7f5a21a9df","Type":"ContainerDied","Data":"adb7490dc70dd48d20b6ba7d3c459b400233149636874e8a3ebb9518882746aa"}
Oct 30 13:00:21.757331 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.757303    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"]
Oct 30 13:00:21.758435 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.758372    5837 scope.go:122] "RemoveContainer" containerID="74d4cc7509b72ac885835487d4de13fd8b464d4a147d657907a85b3d67acab70"
Oct 30 13:00:21.834626 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.834595    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"]
Oct 30 13:00:21.857708 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.857681    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"]
Oct 30 13:00:21.903433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.898270    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"]
Oct 30 13:00:21.936313 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.930650    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dea6ee-29889420-wmh27"]
Oct 30 13:00:21.943342 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:21.939452    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"]
Oct 30 13:00:22.226101 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.225608    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"
Oct 30 13:00:22.230907 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.229939    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"
Oct 30 13:00:22.250530 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.250455    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"
Oct 30 13:00:22.330342 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.330308    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f81fbc0-f46e-4730-8e0d-d44a67f089c8-kube-api-access-xjpbq\" (UniqueName: \"kubernetes.io/projected/2f81fbc0-f46e-4730-8e0d-d44a67f089c8-kube-api-access-xjpbq\") pod \"2f81fbc0-f46e-4730-8e0d-d44a67f089c8\" (UID: \"2f81fbc0-f46e-4730-8e0d-d44a67f089c8\") "
Oct 30 13:00:22.330522 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.330411    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa90322-42c2-45b4-a25a-2d4b68c45164-kube-api-access-bv6pt\" (UniqueName: \"kubernetes.io/projected/0fa90322-42c2-45b4-a25a-2d4b68c45164-kube-api-access-bv6pt\") pod \"0fa90322-42c2-45b4-a25a-2d4b68c45164\" (UID: \"0fa90322-42c2-45b4-a25a-2d4b68c45164\") "
Oct 30 13:00:22.330522 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.330447    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bb0c30-e293-4b79-9e36-d37599110835-kube-api-access-8kfbr\" (UniqueName: \"kubernetes.io/projected/b6bb0c30-e293-4b79-9e36-d37599110835-kube-api-access-8kfbr\") pod \"b6bb0c30-e293-4b79-9e36-d37599110835\" (UID: \"b6bb0c30-e293-4b79-9e36-d37599110835\") "
Oct 30 13:00:22.330522 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.330507    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa90322-42c2-45b4-a25a-2d4b68c45164-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa90322-42c2-45b4-a25a-2d4b68c45164-maas-api-service-ca\") pod \"0fa90322-42c2-45b4-a25a-2d4b68c45164\" (UID: \"0fa90322-42c2-45b4-a25a-2d4b68c45164\") "
Oct 30 13:00:22.330699 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.330554    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f81fbc0-f46e-4730-8e0d-d44a67f089c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f81fbc0-f46e-4730-8e0d-d44a67f089c8-maas-api-service-ca\") pod \"2f81fbc0-f46e-4730-8e0d-d44a67f089c8\" (UID: \"2f81fbc0-f46e-4730-8e0d-d44a67f089c8\") "
Oct 30 13:00:22.332122 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.330856    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bb0c30-e293-4b79-9e36-d37599110835-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bb0c30-e293-4b79-9e36-d37599110835-maas-api-service-ca\") pod \"b6bb0c30-e293-4b79-9e36-d37599110835\" (UID: \"b6bb0c30-e293-4b79-9e36-d37599110835\") "
Oct 30 13:00:22.332122 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.331945    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bb0c30-e293-4b79-9e36-d37599110835-maas-api-service-ca" pod "b6bb0c30-e293-4b79-9e36-d37599110835" (UID: "b6bb0c30-e293-4b79-9e36-d37599110835"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:22.333668 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.332700    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa90322-42c2-45b4-a25a-2d4b68c45164-maas-api-service-ca" pod "0fa90322-42c2-45b4-a25a-2d4b68c45164" (UID: "0fa90322-42c2-45b4-a25a-2d4b68c45164"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:22.333668 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.332949    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f81fbc0-f46e-4730-8e0d-d44a67f089c8-maas-api-service-ca" pod "2f81fbc0-f46e-4730-8e0d-d44a67f089c8" (UID: "2f81fbc0-f46e-4730-8e0d-d44a67f089c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:22.338985 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.338829    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bb0c30-e293-4b79-9e36-d37599110835-kube-api-access-8kfbr" pod "b6bb0c30-e293-4b79-9e36-d37599110835" (UID: "b6bb0c30-e293-4b79-9e36-d37599110835"). InnerVolumeSpecName "kube-api-access-8kfbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:22.338985 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.338883    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa90322-42c2-45b4-a25a-2d4b68c45164-kube-api-access-bv6pt" pod "0fa90322-42c2-45b4-a25a-2d4b68c45164" (UID: "0fa90322-42c2-45b4-a25a-2d4b68c45164"). InnerVolumeSpecName "kube-api-access-bv6pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:22.340132 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.339529    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f81fbc0-f46e-4730-8e0d-d44a67f089c8-kube-api-access-xjpbq" pod "2f81fbc0-f46e-4730-8e0d-d44a67f089c8" (UID: "2f81fbc0-f46e-4730-8e0d-d44a67f089c8"). InnerVolumeSpecName "kube-api-access-xjpbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:22.428324 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.428291    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"
Oct 30 13:00:22.433499 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.433467    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bb0c30-e293-4b79-9e36-d37599110835-maas-api-service-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.433499 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.433494    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjpbq\" (UniqueName: \"kubernetes.io/projected/2f81fbc0-f46e-4730-8e0d-d44a67f089c8-kube-api-access-xjpbq\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.433745 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.433510    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv6pt\" (UniqueName: \"kubernetes.io/projected/0fa90322-42c2-45b4-a25a-2d4b68c45164-kube-api-access-bv6pt\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.433745 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.433525    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kfbr\" (UniqueName: \"kubernetes.io/projected/b6bb0c30-e293-4b79-9e36-d37599110835-kube-api-access-8kfbr\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.433745 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.433538    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa90322-42c2-45b4-a25a-2d4b68c45164-maas-api-service-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.433745 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.433552    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f81fbc0-f46e-4730-8e0d-d44a67f089c8-maas-api-service-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.434740 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.434721    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 13:00:22.592430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.591279    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-credential-socket\") pod \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") "
Oct 30 13:00:22.592430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.591343    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-kube-api-access-4z4hq\" (UniqueName: \"kubernetes.io/projected/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-kube-api-access-4z4hq\") pod \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") "
Oct 30 13:00:22.592430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.591386    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-envoy\") pod \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") "
Oct 30 13:00:22.592430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.591475    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-workload-socket\") pod \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") "
Oct 30 13:00:22.592430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.591524    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71007e75-4683-4704-bf4e-11cd7669f1ba-kube-api-access-5xtzn\" (UniqueName: \"kubernetes.io/projected/71007e75-4683-4704-bf4e-11cd7669f1ba-kube-api-access-5xtzn\") pod \"71007e75-4683-4704-bf4e-11cd7669f1ba\" (UID: \"71007e75-4683-4704-bf4e-11cd7669f1ba\") "
Oct 30 13:00:22.592430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.591564    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71007e75-4683-4704-bf4e-11cd7669f1ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71007e75-4683-4704-bf4e-11cd7669f1ba-maas-api-service-ca\") pod \"71007e75-4683-4704-bf4e-11cd7669f1ba\" (UID: \"71007e75-4683-4704-bf4e-11cd7669f1ba\") "
Oct 30 13:00:22.592430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.591612    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-data\") pod \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") "
Oct 30 13:00:22.592430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.591648    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istiod-ca-cert\") pod \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") "
Oct 30 13:00:22.592430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.591682    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-podinfo\") pod \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") "
Oct 30 13:00:22.592430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.591732    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-token\" (UniqueName: \"kubernetes.io/projected/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-token\") pod \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") "
Oct 30 13:00:22.592430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.591881    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-workload-certs\") pod \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\" (UID: \"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113\") "
Oct 30 13:00:22.657418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.651570    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-workload-certs" pod "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" (UID: "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:22.662109 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.662072    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71007e75-4683-4704-bf4e-11cd7669f1ba-kube-api-access-5xtzn" pod "71007e75-4683-4704-bf4e-11cd7669f1ba" (UID: "71007e75-4683-4704-bf4e-11cd7669f1ba"). InnerVolumeSpecName "kube-api-access-5xtzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:22.663104 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.663076    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-credential-socket" pod "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" (UID: "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:22.696476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.689682    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istiod-ca-cert" pod "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" (UID: "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:22.696476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.694730    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-podinfo" pod "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" (UID: "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:00:22.696476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.694966    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-data" pod "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" (UID: "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:22.696947 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.696865    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-workload-socket" pod "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" (UID: "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:22.697356 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.697330    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71007e75-4683-4704-bf4e-11cd7669f1ba-maas-api-service-ca" pod "71007e75-4683-4704-bf4e-11cd7669f1ba" (UID: "71007e75-4683-4704-bf4e-11cd7669f1ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:22.715803 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.700011    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.716122 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.716102    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xtzn\" (UniqueName: \"kubernetes.io/projected/71007e75-4683-4704-bf4e-11cd7669f1ba-kube-api-access-5xtzn\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.716227 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.716215    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.716321 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.716310    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.716424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.716412    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.716534 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.716521    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.716623 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.716612    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.716891 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.716854    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-kube-api-access-4z4hq" pod "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" (UID: "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113"). InnerVolumeSpecName "kube-api-access-4z4hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:22.723852 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.723793    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-envoy" pod "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" (UID: "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:22.749357 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.749315    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-token" pod "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" (UID: "d9a3fc92-01d1-45c0-ab72-c1eb52ed7113"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:22.770435 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.766763    5837 generic.go:356] "Generic (PLEG): container finished" podID="d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" containerID="4e4f77c9e3572a67ac3665d35bc445bfb6f4e71d412e66fe591ff12cc0ea0684" exitCode=0
Oct 30 13:00:22.770435 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.766872    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5" event={"ID":"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113","Type":"ContainerDied","Data":"4e4f77c9e3572a67ac3665d35bc445bfb6f4e71d412e66fe591ff12cc0ea0684"}
Oct 30 13:00:22.770435 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.766927    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5" event={"ID":"d9a3fc92-01d1-45c0-ab72-c1eb52ed7113","Type":"ContainerDied","Data":"1079b3d6ea5d2b3e475e2b75dea7b9b723ab917dc7ea8f9bad341ac240b48390"}
Oct 30 13:00:22.770435 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.766965    5837 scope.go:122] "RemoveContainer" containerID="4e4f77c9e3572a67ac3665d35bc445bfb6f4e71d412e66fe591ff12cc0ea0684"
Oct 30 13:00:22.770435 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.767081    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"
Oct 30 13:00:22.777437 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.777367    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w" event={"ID":"0fa90322-42c2-45b4-a25a-2d4b68c45164","Type":"ContainerDied","Data":"ecda07bf8ea8aa27e28a3c1b2ec23b176c506bccf51293077ee711bcc8972ed2"}
Oct 30 13:00:22.777588 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.777513    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"
Oct 30 13:00:22.803656 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.800568    5837 scope.go:122] "RemoveContainer" containerID="b92a811e43f026fa57e988fd33f822abda88ba9b84353a6cc22cdbf3738933f9"
Oct 30 13:00:22.819430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.816630    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz" event={"ID":"71007e75-4683-4704-bf4e-11cd7669f1ba","Type":"ContainerDied","Data":"39c7105f5fb6595755b41b9d63e4e8e68ffc4a1bb8780c344b7ca97d47668692"}
Oct 30 13:00:22.819430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.816736    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"
Oct 30 13:00:22.819430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.818484    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z4hq\" (UniqueName: \"kubernetes.io/projected/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-kube-api-access-4z4hq\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.819430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.818510    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.819430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.818527    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71007e75-4683-4704-bf4e-11cd7669f1ba-maas-api-service-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.819430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.818541    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.842110 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.842086    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"
Oct 30 13:00:22.842658 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.842580    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95" event={"ID":"b6bb0c30-e293-4b79-9e36-d37599110835","Type":"ContainerDied","Data":"734a00777edc586d1f0d2831ba64e2a05c6d703ab69736057fc4cc73fbdd189e"}
Oct 30 13:00:22.846526 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.846493    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8" event={"ID":"2f81fbc0-f46e-4730-8e0d-d44a67f089c8","Type":"ContainerDied","Data":"d367c623d83c0f15381cde774308df587058821519db1c292570cc30ac8d699a"}
Oct 30 13:00:22.846673 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.846605    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"
Oct 30 13:00:22.864151 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.863840    5837 scope.go:122] "RemoveContainer" containerID="4e4f77c9e3572a67ac3665d35bc445bfb6f4e71d412e66fe591ff12cc0ea0684"
Oct 30 13:00:22.864289 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:22.864221    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4f77c9e3572a67ac3665d35bc445bfb6f4e71d412e66fe591ff12cc0ea0684\": container with ID starting with 4e4f77c9e3572a67ac3665d35bc445bfb6f4e71d412e66fe591ff12cc0ea0684 not found: ID does not exist" containerID="4e4f77c9e3572a67ac3665d35bc445bfb6f4e71d412e66fe591ff12cc0ea0684"
Oct 30 13:00:22.864289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.864256    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4f77c9e3572a67ac3665d35bc445bfb6f4e71d412e66fe591ff12cc0ea0684"} err="failed to get container status \"4e4f77c9e3572a67ac3665d35bc445bfb6f4e71d412e66fe591ff12cc0ea0684\": rpc error: code = NotFound desc = could not find container \"4e4f77c9e3572a67ac3665d35bc445bfb6f4e71d412e66fe591ff12cc0ea0684\": container with ID starting with 4e4f77c9e3572a67ac3665d35bc445bfb6f4e71d412e66fe591ff12cc0ea0684 not found: ID does not exist"
Oct 30 13:00:22.864289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.864281    5837 scope.go:122] "RemoveContainer" containerID="b92a811e43f026fa57e988fd33f822abda88ba9b84353a6cc22cdbf3738933f9"
Oct 30 13:00:22.865626 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:22.865603    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b92a811e43f026fa57e988fd33f822abda88ba9b84353a6cc22cdbf3738933f9\": container with ID starting with b92a811e43f026fa57e988fd33f822abda88ba9b84353a6cc22cdbf3738933f9 not found: ID does not exist" containerID="b92a811e43f026fa57e988fd33f822abda88ba9b84353a6cc22cdbf3738933f9"
Oct 30 13:00:22.865746 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.865634    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b92a811e43f026fa57e988fd33f822abda88ba9b84353a6cc22cdbf3738933f9"} err="failed to get container status \"b92a811e43f026fa57e988fd33f822abda88ba9b84353a6cc22cdbf3738933f9\": rpc error: code = NotFound desc = could not find container \"b92a811e43f026fa57e988fd33f822abda88ba9b84353a6cc22cdbf3738933f9\": container with ID starting with b92a811e43f026fa57e988fd33f822abda88ba9b84353a6cc22cdbf3738933f9 not found: ID does not exist"
Oct 30 13:00:22.865746 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.865657    5837 scope.go:122] "RemoveContainer" containerID="dc73fb04782cc8a3922825a7188da7cd7b260f9b04c13967ec5ef377f722c654"
Oct 30 13:00:22.921313 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.889561    5837 scope.go:122] "RemoveContainer" containerID="4aa8c2d3ed161cc15bad99a46c0c03ee4b04d03410ccb3da5a471c1e006f32ea"
Oct 30 13:00:22.975812 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.964449    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"]
Oct 30 13:00:22.976250 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.976229    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a590a-29889420-zkd95"]
Oct 30 13:00:22.983449 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:22.983286    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"]
Oct 30 13:00:23.060456 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.057838    5837 scope.go:122] "RemoveContainer" containerID="81b9ce84735ad8138401c2d241090dec95f68ed1aab0eb872400ef258f657e13"
Oct 30 13:00:23.071047 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.071015    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53dcb1-29889420-d5xn8"]
Oct 30 13:00:23.114420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.083658    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"]
Oct 30 13:00:23.145570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.145538    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-92fw5"]
Oct 30 13:00:23.147462 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.147425    5837 scope.go:122] "RemoveContainer" containerID="1d83375388f38e5659278633c8dfb0fe3b08e5a5d0e511338534d1681a8c4cc5"
Oct 30 13:00:23.153886 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.153829    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"]
Oct 30 13:00:23.158754 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.158731    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0a01c-29889420-qqx2w"]
Oct 30 13:00:23.192602 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.183116    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"]
Oct 30 13:00:23.196618 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.196590    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-105228-29889420-l2ppz"]
Oct 30 13:00:23.230552 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.230524    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"
Oct 30 13:00:23.441073 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.440247    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e00d30-b982-4c55-b2a5-984eb00676ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e00d30-b982-4c55-b2a5-984eb00676ef-maas-api-service-ca\") pod \"e2e00d30-b982-4c55-b2a5-984eb00676ef\" (UID: \"e2e00d30-b982-4c55-b2a5-984eb00676ef\") "
Oct 30 13:00:23.441073 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.440503    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e00d30-b982-4c55-b2a5-984eb00676ef-kube-api-access-gllsr\" (UniqueName: \"kubernetes.io/projected/e2e00d30-b982-4c55-b2a5-984eb00676ef-kube-api-access-gllsr\") pod \"e2e00d30-b982-4c55-b2a5-984eb00676ef\" (UID: \"e2e00d30-b982-4c55-b2a5-984eb00676ef\") "
Oct 30 13:00:23.441992 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.441965    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e00d30-b982-4c55-b2a5-984eb00676ef-maas-api-service-ca" pod "e2e00d30-b982-4c55-b2a5-984eb00676ef" (UID: "e2e00d30-b982-4c55-b2a5-984eb00676ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:23.478349 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.478314    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e00d30-b982-4c55-b2a5-984eb00676ef-kube-api-access-gllsr" pod "e2e00d30-b982-4c55-b2a5-984eb00676ef" (UID: "e2e00d30-b982-4c55-b2a5-984eb00676ef"). InnerVolumeSpecName "kube-api-access-gllsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:23.549876 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.549782    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e00d30-b982-4c55-b2a5-984eb00676ef-maas-api-service-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.550145 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.550128    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gllsr\" (UniqueName: \"kubernetes.io/projected/e2e00d30-b982-4c55-b2a5-984eb00676ef-kube-api-access-gllsr\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.953696 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.953612    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa90322-42c2-45b4-a25a-2d4b68c45164" path="/var/lib/kubelet/pods/0fa90322-42c2-45b4-a25a-2d4b68c45164/volumes"
Oct 30 13:00:23.954337 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.954277    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8" path="/var/lib/kubelet/pods/2f81fbc0-f46e-4730-8e0d-d44a67f089c8/volumes"
Oct 30 13:00:23.955037 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.954818    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71007e75-4683-4704-bf4e-11cd7669f1ba" path="/var/lib/kubelet/pods/71007e75-4683-4704-bf4e-11cd7669f1ba/volumes"
Oct 30 13:00:23.955698 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.955680    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84fb8f80-260a-41ee-9df6-3a7f5a21a9df" path="/var/lib/kubelet/pods/84fb8f80-260a-41ee-9df6-3a7f5a21a9df/volumes"
Oct 30 13:00:23.956250 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.956169    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6bb0c30-e293-4b79-9e36-d37599110835" path="/var/lib/kubelet/pods/b6bb0c30-e293-4b79-9e36-d37599110835/volumes"
Oct 30 13:00:23.956630 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.956615    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" path="/var/lib/kubelet/pods/d9a3fc92-01d1-45c0-ab72-c1eb52ed7113/volumes"
Oct 30 13:00:23.962302 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.961204    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf" event={"ID":"e2e00d30-b982-4c55-b2a5-984eb00676ef","Type":"ContainerDied","Data":"6b5faf9a3646d2ddd86f1393b89fb0e989724c81d02b482b0363faeee7f00086"}
Oct 30 13:00:23.962302 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.961266    5837 scope.go:122] "RemoveContainer" containerID="1d938c94fcdfdc12c9aaff3d0898890222aa7962fa87e03a7c4dc142fcd3bcb5"
Oct 30 13:00:23.962302 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:23.961360    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"
Oct 30 13:00:24.039191 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:24.039158    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"]
Oct 30 13:00:24.043111 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:24.043078    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4cd3fb-29889420-22flf"]
Oct 30 13:00:25.146174 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.146142    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"]
Oct 30 13:00:25.146960 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.146943    5837 cpu_manager.go:414] "containerMap: removing container" podUID="71007e75-4683-4704-bf4e-11cd7669f1ba" containerName="cleanup"
Oct 30 13:00:25.147004 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.146962    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71007e75-4683-4704-bf4e-11cd7669f1ba" containerName="cleanup"
Oct 30 13:00:25.147004 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.146971    5837 cpu_manager.go:414] "containerMap: removing container" podUID="84fb8f80-260a-41ee-9df6-3a7f5a21a9df" containerName="cleanup"
Oct 30 13:00:25.147004 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.146977    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fb8f80-260a-41ee-9df6-3a7f5a21a9df" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147004    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0fa90322-42c2-45b4-a25a-2d4b68c45164" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147010    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa90322-42c2-45b4-a25a-2d4b68c45164" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147025    5837 cpu_manager.go:414] "containerMap: removing container" podUID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147030    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147036    5837 cpu_manager.go:414] "containerMap: removing container" podUID="e2e00d30-b982-4c55-b2a5-984eb00676ef" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147040    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e00d30-b982-4c55-b2a5-984eb00676ef" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147048    5837 cpu_manager.go:414] "containerMap: removing container" podUID="e2e00d30-b982-4c55-b2a5-984eb00676ef" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147052    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e00d30-b982-4c55-b2a5-984eb00676ef" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147063    5837 cpu_manager.go:414] "containerMap: removing container" podUID="739294c5-2be4-4958-891a-bf63f2081bef" containerName="revoke-keys"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147068    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739294c5-2be4-4958-891a-bf63f2081bef" containerName="revoke-keys"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147077    5837 cpu_manager.go:414] "containerMap: removing container" podUID="84fb8f80-260a-41ee-9df6-3a7f5a21a9df" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147081    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fb8f80-260a-41ee-9df6-3a7f5a21a9df" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147087    5837 cpu_manager.go:414] "containerMap: removing container" podUID="71007e75-4683-4704-bf4e-11cd7669f1ba" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147091    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71007e75-4683-4704-bf4e-11cd7669f1ba" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147103    5837 cpu_manager.go:414] "containerMap: removing container" podUID="d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" containerName="istio-proxy"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147107    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" containerName="istio-proxy"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147114    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0fa90322-42c2-45b4-a25a-2d4b68c45164" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147119    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa90322-42c2-45b4-a25a-2d4b68c45164" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147127    5837 cpu_manager.go:414] "containerMap: removing container" podUID="84fb8f80-260a-41ee-9df6-3a7f5a21a9df" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147131    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fb8f80-260a-41ee-9df6-3a7f5a21a9df" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147140    5837 cpu_manager.go:414] "containerMap: removing container" podUID="d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" containerName="istio-proxy"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147144    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" containerName="istio-proxy"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147153    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0fa90322-42c2-45b4-a25a-2d4b68c45164" containerName="cleanup"
Oct 30 13:00:25.147146 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147157    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa90322-42c2-45b4-a25a-2d4b68c45164" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147173    5837 cpu_manager.go:414] "containerMap: removing container" podUID="b6bb0c30-e293-4b79-9e36-d37599110835" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147180    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bb0c30-e293-4b79-9e36-d37599110835" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147193    5837 cpu_manager.go:414] "containerMap: removing container" podUID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147200    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147207    5837 cpu_manager.go:414] "containerMap: removing container" podUID="b6bb0c30-e293-4b79-9e36-d37599110835" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147211    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bb0c30-e293-4b79-9e36-d37599110835" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147220    5837 cpu_manager.go:414] "containerMap: removing container" podUID="e2e00d30-b982-4c55-b2a5-984eb00676ef" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147224    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e00d30-b982-4c55-b2a5-984eb00676ef" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147234    5837 cpu_manager.go:414] "containerMap: removing container" podUID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147238    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147245    5837 cpu_manager.go:414] "containerMap: removing container" podUID="b6bb0c30-e293-4b79-9e36-d37599110835" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147249    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bb0c30-e293-4b79-9e36-d37599110835" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147368    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e00d30-b982-4c55-b2a5-984eb00676ef" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147376    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147383    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bb0c30-e293-4b79-9e36-d37599110835" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147415    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="71007e75-4683-4704-bf4e-11cd7669f1ba" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147421    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fb8f80-260a-41ee-9df6-3a7f5a21a9df" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147426    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa90322-42c2-45b4-a25a-2d4b68c45164" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147433    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147438    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e00d30-b982-4c55-b2a5-984eb00676ef" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147443    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e00d30-b982-4c55-b2a5-984eb00676ef" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147450    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="739294c5-2be4-4958-891a-bf63f2081bef" containerName="revoke-keys"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147456    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fb8f80-260a-41ee-9df6-3a7f5a21a9df" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147461    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="71007e75-4683-4704-bf4e-11cd7669f1ba" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147468    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" containerName="istio-proxy"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147473    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fb8f80-260a-41ee-9df6-3a7f5a21a9df" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147480    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a3fc92-01d1-45c0-ab72-c1eb52ed7113" containerName="istio-proxy"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147485    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="71007e75-4683-4704-bf4e-11cd7669f1ba" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147491    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa90322-42c2-45b4-a25a-2d4b68c45164" containerName="cleanup"
Oct 30 13:00:25.147908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.147500    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bb0c30-e293-4b79-9e36-d37599110835" containerName="cleanup"
Oct 30 13:00:25.152564 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.152547    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"
Oct 30 13:00:25.168811 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.168783    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"]
Oct 30 13:00:25.181636 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.181591    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:00:25.181636 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.181622    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:00:25.182789 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.182765    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cdd1d1c-6p9zf\" (UID: \"73361c80-e0ae-4807-ae3c-e5a64d2f18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"
Oct 30 13:00:25.182927 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.182909    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cdd1d1c-6p9zf\" (UID: \"73361c80-e0ae-4807-ae3c-e5a64d2f18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"
Oct 30 13:00:25.183005 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.182985    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z824r\" (UniqueName: \"kubernetes.io/projected/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-kube-api-access-z824r\") pod \"maas-api-revoke-sub-3cdd1d1c-6p9zf\" (UID: \"73361c80-e0ae-4807-ae3c-e5a64d2f18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"
Oct 30 13:00:25.256357 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.256324    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:00:25.256357 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.256359    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:00:25.304418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.300317    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cdd1d1c-6p9zf\" (UID: \"73361c80-e0ae-4807-ae3c-e5a64d2f18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"
Oct 30 13:00:25.304418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.302751    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cdd1d1c-6p9zf\" (UID: \"73361c80-e0ae-4807-ae3c-e5a64d2f18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"
Oct 30 13:00:25.304418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.302821    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cdd1d1c-6p9zf\" (UID: \"73361c80-e0ae-4807-ae3c-e5a64d2f18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"
Oct 30 13:00:25.314420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.313231    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z824r\" (UniqueName: \"kubernetes.io/projected/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-kube-api-access-z824r\") pod \"maas-api-revoke-sub-3cdd1d1c-6p9zf\" (UID: \"73361c80-e0ae-4807-ae3c-e5a64d2f18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"
Oct 30 13:00:25.357667 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.356367    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cdd1d1c-6p9zf\" (UID: \"73361c80-e0ae-4807-ae3c-e5a64d2f18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"
Oct 30 13:00:25.357667 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.357183    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z824r\" (UniqueName: \"kubernetes.io/projected/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-kube-api-access-z824r\") pod \"maas-api-revoke-sub-3cdd1d1c-6p9zf\" (UID: \"73361c80-e0ae-4807-ae3c-e5a64d2f18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"
Oct 30 13:00:25.470900 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.470670    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"
Oct 30 13:00:25.634186 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.633518    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e00d30-b982-4c55-b2a5-984eb00676ef" path="/var/lib/kubelet/pods/e2e00d30-b982-4c55-b2a5-984eb00676ef/volumes"
Oct 30 13:00:25.864194 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:25.864164    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"]
Oct 30 13:00:25.869474 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:00:25.869446    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73361c80_e0ae_4807_ae3c_e5a64d2f18fe.slice/crio-049deec79410df9606f7e2fb815eb3bb77e0781672ddc56affee73c6e07b8aa0 WatchSource:0}: Error finding container 049deec79410df9606f7e2fb815eb3bb77e0781672ddc56affee73c6e07b8aa0: Status 404 returned error can't find the container with id 049deec79410df9606f7e2fb815eb3bb77e0781672ddc56affee73c6e07b8aa0
Oct 30 13:00:26.010537 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:26.010496    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf" event={"ID":"73361c80-e0ae-4807-ae3c-e5a64d2f18fe","Type":"ContainerStarted","Data":"049deec79410df9606f7e2fb815eb3bb77e0781672ddc56affee73c6e07b8aa0"}
Oct 30 13:00:26.566594 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:26.566553    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:00:26.567040 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:26.566623    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:27.020882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:27.020845    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf" event={"ID":"73361c80-e0ae-4807-ae3c-e5a64d2f18fe","Type":"ContainerStarted","Data":"c493db9af9d2cbcf9223d8d6e59f0671a34b02485fc6094c235afca851eed2bd"}
Oct 30 13:00:27.061801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:27.061734    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf" podStartSLOduration=2.061714203 podStartE2EDuration="2.061714203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:27.059853054 +0000 UTC m=+1415.584137275" watchObservedRunningTime="2026-10-30 13:00:27.061714203 +0000 UTC m=+1415.585998423"
Oct 30 13:00:28.143518 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:28.143224    5837 generic.go:356] "Generic (PLEG): container finished" podID="73361c80-e0ae-4807-ae3c-e5a64d2f18fe" containerID="c493db9af9d2cbcf9223d8d6e59f0671a34b02485fc6094c235afca851eed2bd" exitCode=0
Oct 30 13:00:28.143518 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:28.143308    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf" event={"ID":"73361c80-e0ae-4807-ae3c-e5a64d2f18fe","Type":"ContainerDied","Data":"c493db9af9d2cbcf9223d8d6e59f0671a34b02485fc6094c235afca851eed2bd"}
Oct 30 13:00:29.339814 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:29.339784    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"
Oct 30 13:00:29.433064 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:29.432982    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-kube-api-access-z824r\" (UniqueName: \"kubernetes.io/projected/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-kube-api-access-z824r\") pod \"73361c80-e0ae-4807-ae3c-e5a64d2f18fe\" (UID: \"73361c80-e0ae-4807-ae3c-e5a64d2f18fe\") "
Oct 30 13:00:29.433330 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:29.433298    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-maas-api-openshift-ca\") pod \"73361c80-e0ae-4807-ae3c-e5a64d2f18fe\" (UID: \"73361c80-e0ae-4807-ae3c-e5a64d2f18fe\") "
Oct 30 13:00:29.433765 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:29.433650    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-maas-api-xks-ca\") pod \"73361c80-e0ae-4807-ae3c-e5a64d2f18fe\" (UID: \"73361c80-e0ae-4807-ae3c-e5a64d2f18fe\") "
Oct 30 13:00:29.437220 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:29.435292    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-maas-api-openshift-ca" pod "73361c80-e0ae-4807-ae3c-e5a64d2f18fe" (UID: "73361c80-e0ae-4807-ae3c-e5a64d2f18fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:29.437220 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:29.435655    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:29.445557 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:29.443917    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-maas-api-xks-ca" pod "73361c80-e0ae-4807-ae3c-e5a64d2f18fe" (UID: "73361c80-e0ae-4807-ae3c-e5a64d2f18fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:00:29.456585 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:29.455276    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-kube-api-access-z824r" pod "73361c80-e0ae-4807-ae3c-e5a64d2f18fe" (UID: "73361c80-e0ae-4807-ae3c-e5a64d2f18fe"). InnerVolumeSpecName "kube-api-access-z824r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:29.536851 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:29.536813    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:29.536851 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:29.536844    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z824r\" (UniqueName: \"kubernetes.io/projected/73361c80-e0ae-4807-ae3c-e5a64d2f18fe-kube-api-access-z824r\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:30.028868 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:30.028839    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"]
Oct 30 13:00:30.034441 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:30.034408    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d2ef82b-g8qxj"]
Oct 30 13:00:30.164534 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:30.164437    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"
Oct 30 13:00:30.164820 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:30.164442    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf" event={"ID":"73361c80-e0ae-4807-ae3c-e5a64d2f18fe","Type":"ContainerDied","Data":"049deec79410df9606f7e2fb815eb3bb77e0781672ddc56affee73c6e07b8aa0"}
Oct 30 13:00:30.164953 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:30.164869    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="049deec79410df9606f7e2fb815eb3bb77e0781672ddc56affee73c6e07b8aa0"
Oct 30 13:00:30.166595 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:30.166571    5837 generic.go:356] "Generic (PLEG): container finished" podID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerID="c8da97fb89e6d59c66ff8b4144070c3d0054c9a22fdf1c723fdf9d8c888ad50a" exitCode=0
Oct 30 13:00:30.166726 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:30.166627    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" event={"ID":"d3aed56e-a69c-46a1-97e6-ea012d99e593","Type":"ContainerDied","Data":"c8da97fb89e6d59c66ff8b4144070c3d0054c9a22fdf1c723fdf9d8c888ad50a"}
Oct 30 13:00:30.167106 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:30.167086    5837 scope.go:122] "RemoveContainer" containerID="c8da97fb89e6d59c66ff8b4144070c3d0054c9a22fdf1c723fdf9d8c888ad50a"
Oct 30 13:00:30.169013 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:30.168978    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:30.169099 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:30.169077    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:30.169155 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:30.169139    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:31.177640 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:31.177598    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" event={"ID":"d3aed56e-a69c-46a1-97e6-ea012d99e593","Type":"ContainerStarted","Data":"9147103a9b30857ae6dc9eee00084a44fd7616d425de4e8b5284008c3f87a216"}
Oct 30 13:00:31.390421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:31.384908    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 13:00:31.390421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:31.385507    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 13:00:31.390421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:31.385578    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 13:00:31.628894 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:31.628848    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="747e76a9-41d7-4942-a33f-53aa5fd53507" path="/var/lib/kubelet/pods/747e76a9-41d7-4942-a33f-53aa5fd53507/volumes"
Oct 30 13:00:32.358044 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:32.358009    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 13:00:32.358492 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:32.358088    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 13:00:33.359732 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:33.359688    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 13:00:33.360265 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:33.359767    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 13:00:33.481983 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:33.481934    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"]
Oct 30 13:00:33.482188 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:33.482131    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr" podUID="b8233dac-3352-487b-97f1-9464e07860b7" containerName="payload-processing" containerID="cri-o://0c71b5d3091f8e725f25890e0ce989e2b4c69aaee32b7d290aa71dce0c7988b9" gracePeriod=30
Oct 30 13:00:33.515420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:33.514546    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"]
Oct 30 13:00:33.515999 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:33.515939    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx" podUID="1ef16201-4106-444c-8a76-7fb8f437f137" containerName="payload-pre-processing" containerID="cri-o://39f5d7b16dd7b5d78f3d981fb4bdf8bf5ec8ac960d8d09bc968c556a51a0f9f8" gracePeriod=30
Oct 30 13:00:34.192999 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.154612    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"
Oct 30 13:00:34.192999 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.160947    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"]
Oct 30 13:00:34.192999 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.166978    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2bfb1b-xtwmc"]
Oct 30 13:00:34.275240 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.268906    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8233dac-3352-487b-97f1-9464e07860b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8233dac-3352-487b-97f1-9464e07860b7-plugins-config-volume\") pod \"b8233dac-3352-487b-97f1-9464e07860b7\" (UID: \"b8233dac-3352-487b-97f1-9464e07860b7\") "
Oct 30 13:00:34.275240 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.275018    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8233dac-3352-487b-97f1-9464e07860b7-plugins-config-volume" pod "b8233dac-3352-487b-97f1-9464e07860b7" (UID: "b8233dac-3352-487b-97f1-9464e07860b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:34.277083 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.276560    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8233dac-3352-487b-97f1-9464e07860b7-kube-api-access-n54pm\" (UniqueName: \"kubernetes.io/projected/b8233dac-3352-487b-97f1-9464e07860b7-kube-api-access-n54pm\") pod \"b8233dac-3352-487b-97f1-9464e07860b7\" (UID: \"b8233dac-3352-487b-97f1-9464e07860b7\") "
Oct 30 13:00:34.277345 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.277320    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8233dac-3352-487b-97f1-9464e07860b7-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:34.292672 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.280969    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8233dac-3352-487b-97f1-9464e07860b7-kube-api-access-n54pm" pod "b8233dac-3352-487b-97f1-9464e07860b7" (UID: "b8233dac-3352-487b-97f1-9464e07860b7"). InnerVolumeSpecName "kube-api-access-n54pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:34.292672 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.284719    5837 generic.go:356] "Generic (PLEG): container finished" podID="b8233dac-3352-487b-97f1-9464e07860b7" containerID="0c71b5d3091f8e725f25890e0ce989e2b4c69aaee32b7d290aa71dce0c7988b9" exitCode=0
Oct 30 13:00:34.292672 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.284911    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr" event={"ID":"b8233dac-3352-487b-97f1-9464e07860b7","Type":"ContainerDied","Data":"0c71b5d3091f8e725f25890e0ce989e2b4c69aaee32b7d290aa71dce0c7988b9"}
Oct 30 13:00:34.292672 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.284962    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr" event={"ID":"b8233dac-3352-487b-97f1-9464e07860b7","Type":"ContainerDied","Data":"3a2ba6f669f02e30e92dbb37a67a4232d2c2d73f447905c56dc7b28590680d2c"}
Oct 30 13:00:34.292672 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.284990    5837 scope.go:122] "RemoveContainer" containerID="0c71b5d3091f8e725f25890e0ce989e2b4c69aaee32b7d290aa71dce0c7988b9"
Oct 30 13:00:34.292672 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.285150    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"
Oct 30 13:00:34.303592 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.299628    5837 generic.go:356] "Generic (PLEG): container finished" podID="1ef16201-4106-444c-8a76-7fb8f437f137" containerID="39f5d7b16dd7b5d78f3d981fb4bdf8bf5ec8ac960d8d09bc968c556a51a0f9f8" exitCode=0
Oct 30 13:00:34.303592 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.299883    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx" event={"ID":"1ef16201-4106-444c-8a76-7fb8f437f137","Type":"ContainerDied","Data":"39f5d7b16dd7b5d78f3d981fb4bdf8bf5ec8ac960d8d09bc968c556a51a0f9f8"}
Oct 30 13:00:34.341717 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.341688    5837 scope.go:122] "RemoveContainer" containerID="0c71b5d3091f8e725f25890e0ce989e2b4c69aaee32b7d290aa71dce0c7988b9"
Oct 30 13:00:34.342165 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:34.342132    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c71b5d3091f8e725f25890e0ce989e2b4c69aaee32b7d290aa71dce0c7988b9\": container with ID starting with 0c71b5d3091f8e725f25890e0ce989e2b4c69aaee32b7d290aa71dce0c7988b9 not found: ID does not exist" containerID="0c71b5d3091f8e725f25890e0ce989e2b4c69aaee32b7d290aa71dce0c7988b9"
Oct 30 13:00:34.342277 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.342168    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c71b5d3091f8e725f25890e0ce989e2b4c69aaee32b7d290aa71dce0c7988b9"} err="failed to get container status \"0c71b5d3091f8e725f25890e0ce989e2b4c69aaee32b7d290aa71dce0c7988b9\": rpc error: code = NotFound desc = could not find container \"0c71b5d3091f8e725f25890e0ce989e2b4c69aaee32b7d290aa71dce0c7988b9\": container with ID starting with 0c71b5d3091f8e725f25890e0ce989e2b4c69aaee32b7d290aa71dce0c7988b9 not found: ID does not exist"
Oct 30 13:00:34.352092 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.352062    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"
Oct 30 13:00:34.363124 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.362947    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 13:00:34.363124 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.363004    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 13:00:34.378641 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.378557    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n54pm\" (UniqueName: \"kubernetes.io/projected/b8233dac-3352-487b-97f1-9464e07860b7-kube-api-access-n54pm\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:34.382715 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.382686    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"]
Oct 30 13:00:34.423271 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.423240    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2bfb1b-79ddc47b6d-m5wvr"]
Oct 30 13:00:34.480010 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.479985    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef16201-4106-444c-8a76-7fb8f437f137-kube-api-access-swz9m\" (UniqueName: \"kubernetes.io/projected/1ef16201-4106-444c-8a76-7fb8f437f137-kube-api-access-swz9m\") pod \"1ef16201-4106-444c-8a76-7fb8f437f137\" (UID: \"1ef16201-4106-444c-8a76-7fb8f437f137\") "
Oct 30 13:00:34.480170 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.480026    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef16201-4106-444c-8a76-7fb8f437f137-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef16201-4106-444c-8a76-7fb8f437f137-plugins-config-volume\") pod \"1ef16201-4106-444c-8a76-7fb8f437f137\" (UID: \"1ef16201-4106-444c-8a76-7fb8f437f137\") "
Oct 30 13:00:34.480491 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.480462    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef16201-4106-444c-8a76-7fb8f437f137-plugins-config-volume" pod "1ef16201-4106-444c-8a76-7fb8f437f137" (UID: "1ef16201-4106-444c-8a76-7fb8f437f137"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:34.480907 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.480890    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef16201-4106-444c-8a76-7fb8f437f137-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:34.482682 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.482656    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef16201-4106-444c-8a76-7fb8f437f137-kube-api-access-swz9m" pod "1ef16201-4106-444c-8a76-7fb8f437f137" (UID: "1ef16201-4106-444c-8a76-7fb8f437f137"). InnerVolumeSpecName "kube-api-access-swz9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:34.582679 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:34.582649    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swz9m\" (UniqueName: \"kubernetes.io/projected/1ef16201-4106-444c-8a76-7fb8f437f137-kube-api-access-swz9m\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:35.317460 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.317430    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"
Oct 30 13:00:35.317668 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.317433    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx" event={"ID":"1ef16201-4106-444c-8a76-7fb8f437f137","Type":"ContainerDied","Data":"0a553a0278d2780997070886231cb5c00d1af1aed509a826ec8c986ec7c1bd0b"}
Oct 30 13:00:35.317668 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.317572    5837 scope.go:122] "RemoveContainer" containerID="39f5d7b16dd7b5d78f3d981fb4bdf8bf5ec8ac960d8d09bc968c556a51a0f9f8"
Oct 30 13:00:35.320923 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.320901    5837 generic.go:356] "Generic (PLEG): container finished" podID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" containerID="637c02b83950688e877e18e8245863ba25f1074f2b2c0589ec1359c9a33a4dc3" exitCode=0
Oct 30 13:00:35.321047 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.320971    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx" event={"ID":"1eb8b513-9fce-4ddd-9555-9a61cbe430dc","Type":"ContainerDied","Data":"637c02b83950688e877e18e8245863ba25f1074f2b2c0589ec1359c9a33a4dc3"}
Oct 30 13:00:35.321350 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.321336    5837 scope.go:122] "RemoveContainer" containerID="637c02b83950688e877e18e8245863ba25f1074f2b2c0589ec1359c9a33a4dc3"
Oct 30 13:00:35.323135 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.323100    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:35.323213 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.323190    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:35.323253 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.323237    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:35.358000 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.357975    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 13:00:35.358109 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.358025    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 13:00:35.368743 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.368717    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"]
Oct 30 13:00:35.412448 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.376432    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-776b47c7b7-nqqcx"]
Oct 30 13:00:35.621695 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.621609    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef16201-4106-444c-8a76-7fb8f437f137" path="/var/lib/kubelet/pods/1ef16201-4106-444c-8a76-7fb8f437f137/volumes"
Oct 30 13:00:35.622112 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.622098    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739294c5-2be4-4958-891a-bf63f2081bef" path="/var/lib/kubelet/pods/739294c5-2be4-4958-891a-bf63f2081bef/volumes"
Oct 30 13:00:35.622505 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:35.622492    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8233dac-3352-487b-97f1-9464e07860b7" path="/var/lib/kubelet/pods/b8233dac-3352-487b-97f1-9464e07860b7/volumes"
Oct 30 13:00:36.394795 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:36.392772    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 13:00:36.394795 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:36.392845    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 13:00:36.402672 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:36.402542    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx" event={"ID":"1eb8b513-9fce-4ddd-9555-9a61cbe430dc","Type":"ContainerStarted","Data":"f4efac654666cfd62636cf6d5b89ca8778c5f4a6675463bd5df070579054bd17"}
Oct 30 13:00:36.945735 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:36.945699    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 13:00:36.947334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:36.947306    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 13:00:36.947521 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:36.947374    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx" podUID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 13:00:37.362616 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:37.362582    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 13:00:37.362888 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:37.362862    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 13:00:37.363647 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:37.363627    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 13:00:37.945508 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:37.945474    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 13:00:37.946020 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:37.945532    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx" podUID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 13:00:38.945843 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:38.945807    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 13:00:38.946259 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:38.945868    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx" podUID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 13:00:39.947990 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:39.947935    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 13:00:39.948627 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:39.948549    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx" podUID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 13:00:40.773565 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:40.773533    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"]
Oct 30 13:00:40.773767 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:40.773745    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr" podUID="a6163d41-2082-46bd-802d-38271a473e7f" containerName="istio-proxy" containerID="cri-o://67800d8309f8c4a7bb5120ebe83730e14ac2aae038bee7c02ef90221643da05c" gracePeriod=30
Oct 30 13:00:40.816694 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:40.816660    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"]
Oct 30 13:00:40.817220 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:40.817168    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm" podUID="1d739dd4-b062-4a32-a496-637ee6476b02" containerName="istio-proxy" containerID="cri-o://971a01975f4a5a50e0206c16b3bfec554fa872d6363105324136b718fa902b10" gracePeriod=30
Oct 30 13:00:40.950420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:40.947794    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 13:00:40.950420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:40.947880    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx" podUID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 13:00:41.071480 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:41.070511    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 13:00:41.945289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:41.945257    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 13:00:41.945512 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:41.945308    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx" podUID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 13:00:42.949680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:42.949651    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 13:00:42.951017 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:42.950998    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 13:00:45.096301 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:45.096258    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:00:45.921908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.921878    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"]
Oct 30 13:00:45.922614 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.922594    5837 cpu_manager.go:414] "containerMap: removing container" podUID="71007e75-4683-4704-bf4e-11cd7669f1ba" containerName="cleanup"
Oct 30 13:00:45.922614 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.922612    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71007e75-4683-4704-bf4e-11cd7669f1ba" containerName="cleanup"
Oct 30 13:00:45.922805 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.922635    5837 cpu_manager.go:414] "containerMap: removing container" podUID="b8233dac-3352-487b-97f1-9464e07860b7" containerName="payload-processing"
Oct 30 13:00:45.922805 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.922642    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8233dac-3352-487b-97f1-9464e07860b7" containerName="payload-processing"
Oct 30 13:00:45.922805 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.922655    5837 cpu_manager.go:414] "containerMap: removing container" podUID="73361c80-e0ae-4807-ae3c-e5a64d2f18fe" containerName="revoke-keys"
Oct 30 13:00:45.922805 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.922662    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73361c80-e0ae-4807-ae3c-e5a64d2f18fe" containerName="revoke-keys"
Oct 30 13:00:45.922805 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.922688    5837 cpu_manager.go:414] "containerMap: removing container" podUID="1ef16201-4106-444c-8a76-7fb8f437f137" containerName="payload-pre-processing"
Oct 30 13:00:45.922805 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.922695    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef16201-4106-444c-8a76-7fb8f437f137" containerName="payload-pre-processing"
Oct 30 13:00:45.923107 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.922881    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef16201-4106-444c-8a76-7fb8f437f137" containerName="payload-pre-processing"
Oct 30 13:00:45.923107 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.922908    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f81fbc0-f46e-4730-8e0d-d44a67f089c8" containerName="cleanup"
Oct 30 13:00:45.923107 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.922918    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bb0c30-e293-4b79-9e36-d37599110835" containerName="cleanup"
Oct 30 13:00:45.923107 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.922930    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa90322-42c2-45b4-a25a-2d4b68c45164" containerName="cleanup"
Oct 30 13:00:45.923107 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.922942    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8233dac-3352-487b-97f1-9464e07860b7" containerName="payload-processing"
Oct 30 13:00:45.923107 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.922952    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="73361c80-e0ae-4807-ae3c-e5a64d2f18fe" containerName="revoke-keys"
Oct 30 13:00:45.925930 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.925910    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"
Oct 30 13:00:45.945042 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.945015    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"]
Oct 30 13:00:45.955426 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.950345    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:00:45.955426 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.950500    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:00:45.977375 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.977146    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83015f82-5f7e-4a14-9d1e-f9472c34badd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz\" (UID: \"83015f82-5f7e-4a14-9d1e-f9472c34badd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"
Oct 30 13:00:45.977375 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.977304    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83015f82-5f7e-4a14-9d1e-f9472c34badd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz\" (UID: \"83015f82-5f7e-4a14-9d1e-f9472c34badd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"
Oct 30 13:00:45.977375 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.977350    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gn6x\" (UniqueName: \"kubernetes.io/projected/83015f82-5f7e-4a14-9d1e-f9472c34badd-kube-api-access-2gn6x\") pod \"maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz\" (UID: \"83015f82-5f7e-4a14-9d1e-f9472c34badd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"
Oct 30 13:00:45.993878 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.993853    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:00:45.993878 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:45.993855    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:00:46.079229 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.079188    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83015f82-5f7e-4a14-9d1e-f9472c34badd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz\" (UID: \"83015f82-5f7e-4a14-9d1e-f9472c34badd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"
Oct 30 13:00:46.079465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.079332    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83015f82-5f7e-4a14-9d1e-f9472c34badd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz\" (UID: \"83015f82-5f7e-4a14-9d1e-f9472c34badd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"
Oct 30 13:00:46.079465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.079379    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gn6x\" (UniqueName: \"kubernetes.io/projected/83015f82-5f7e-4a14-9d1e-f9472c34badd-kube-api-access-2gn6x\") pod \"maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz\" (UID: \"83015f82-5f7e-4a14-9d1e-f9472c34badd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"
Oct 30 13:00:46.080382 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.080331    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83015f82-5f7e-4a14-9d1e-f9472c34badd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz\" (UID: \"83015f82-5f7e-4a14-9d1e-f9472c34badd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"
Oct 30 13:00:46.082346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.082321    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83015f82-5f7e-4a14-9d1e-f9472c34badd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz\" (UID: \"83015f82-5f7e-4a14-9d1e-f9472c34badd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"
Oct 30 13:00:46.098196 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.098177    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gn6x\" (UniqueName: \"kubernetes.io/projected/83015f82-5f7e-4a14-9d1e-f9472c34badd-kube-api-access-2gn6x\") pod \"maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz\" (UID: \"83015f82-5f7e-4a14-9d1e-f9472c34badd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"
Oct 30 13:00:46.120379 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.120357    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 13:00:46.180115 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.180088    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 13:00:46.180622 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.180602    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-workload-socket\") pod \"a6163d41-2082-46bd-802d-38271a473e7f\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") "
Oct 30 13:00:46.180726 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.180709    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-credential-socket\") pod \"a6163d41-2082-46bd-802d-38271a473e7f\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") "
Oct 30 13:00:46.180787 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.180761    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-istio-envoy\") pod \"a6163d41-2082-46bd-802d-38271a473e7f\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") "
Oct 30 13:00:46.180858 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.180840    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6163d41-2082-46bd-802d-38271a473e7f-istio-token\" (UniqueName: \"kubernetes.io/projected/a6163d41-2082-46bd-802d-38271a473e7f-istio-token\") pod \"a6163d41-2082-46bd-802d-38271a473e7f\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") "
Oct 30 13:00:46.180976 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.180956    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-credential-socket" pod "a6163d41-2082-46bd-802d-38271a473e7f" (UID: "a6163d41-2082-46bd-802d-38271a473e7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:46.180976 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.180971    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-workload-certs\") pod \"a6163d41-2082-46bd-802d-38271a473e7f\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") "
Oct 30 13:00:46.181087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.181010    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6163d41-2082-46bd-802d-38271a473e7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6163d41-2082-46bd-802d-38271a473e7f-istio-podinfo\") pod \"a6163d41-2082-46bd-802d-38271a473e7f\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") "
Oct 30 13:00:46.181087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.181058    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-istio-data\") pod \"a6163d41-2082-46bd-802d-38271a473e7f\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") "
Oct 30 13:00:46.181193 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.181106    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6163d41-2082-46bd-802d-38271a473e7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6163d41-2082-46bd-802d-38271a473e7f-istiod-ca-cert\") pod \"a6163d41-2082-46bd-802d-38271a473e7f\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") "
Oct 30 13:00:46.181193 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.181162    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6163d41-2082-46bd-802d-38271a473e7f-kube-api-access-knfnm\" (UniqueName: \"kubernetes.io/projected/a6163d41-2082-46bd-802d-38271a473e7f-kube-api-access-knfnm\") pod \"a6163d41-2082-46bd-802d-38271a473e7f\" (UID: \"a6163d41-2082-46bd-802d-38271a473e7f\") "
Oct 30 13:00:46.182101 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.181903    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.182101 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.181912    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-workload-certs" pod "a6163d41-2082-46bd-802d-38271a473e7f" (UID: "a6163d41-2082-46bd-802d-38271a473e7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:46.182101 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.181842    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6163d41-2082-46bd-802d-38271a473e7f-istiod-ca-cert" pod "a6163d41-2082-46bd-802d-38271a473e7f" (UID: "a6163d41-2082-46bd-802d-38271a473e7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:46.182101 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.182014    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-istio-data" pod "a6163d41-2082-46bd-802d-38271a473e7f" (UID: "a6163d41-2082-46bd-802d-38271a473e7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:46.182101 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.182095    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-workload-socket" pod "a6163d41-2082-46bd-802d-38271a473e7f" (UID: "a6163d41-2082-46bd-802d-38271a473e7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:46.184429 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.184380    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6163d41-2082-46bd-802d-38271a473e7f-istio-token" pod "a6163d41-2082-46bd-802d-38271a473e7f" (UID: "a6163d41-2082-46bd-802d-38271a473e7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:46.184429 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.184423    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6163d41-2082-46bd-802d-38271a473e7f-kube-api-access-knfnm" pod "a6163d41-2082-46bd-802d-38271a473e7f" (UID: "a6163d41-2082-46bd-802d-38271a473e7f"). InnerVolumeSpecName "kube-api-access-knfnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:46.184686 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.184663    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6163d41-2082-46bd-802d-38271a473e7f-istio-podinfo" pod "a6163d41-2082-46bd-802d-38271a473e7f" (UID: "a6163d41-2082-46bd-802d-38271a473e7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:00:46.187071 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.187043    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-istio-envoy" pod "a6163d41-2082-46bd-802d-38271a473e7f" (UID: "a6163d41-2082-46bd-802d-38271a473e7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:46.244247 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.244219    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"
Oct 30 13:00:46.282387 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.282361    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-istio-envoy\") pod \"1d739dd4-b062-4a32-a496-637ee6476b02\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") "
Oct 30 13:00:46.282572 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.282474    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d739dd4-b062-4a32-a496-637ee6476b02-istio-token\" (UniqueName: \"kubernetes.io/projected/1d739dd4-b062-4a32-a496-637ee6476b02-istio-token\") pod \"1d739dd4-b062-4a32-a496-637ee6476b02\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") "
Oct 30 13:00:46.282638 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.282624    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-credential-socket\") pod \"1d739dd4-b062-4a32-a496-637ee6476b02\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") "
Oct 30 13:00:46.282700 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.282664    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d739dd4-b062-4a32-a496-637ee6476b02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d739dd4-b062-4a32-a496-637ee6476b02-istiod-ca-cert\") pod \"1d739dd4-b062-4a32-a496-637ee6476b02\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") "
Oct 30 13:00:46.282768 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.282741    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-workload-certs\") pod \"1d739dd4-b062-4a32-a496-637ee6476b02\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") "
Oct 30 13:00:46.282822 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.282777    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-workload-socket\") pod \"1d739dd4-b062-4a32-a496-637ee6476b02\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") "
Oct 30 13:00:46.282822 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.282816    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d739dd4-b062-4a32-a496-637ee6476b02-kube-api-access-jvcmg\" (UniqueName: \"kubernetes.io/projected/1d739dd4-b062-4a32-a496-637ee6476b02-kube-api-access-jvcmg\") pod \"1d739dd4-b062-4a32-a496-637ee6476b02\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") "
Oct 30 13:00:46.282938 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.282924    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-istio-data\") pod \"1d739dd4-b062-4a32-a496-637ee6476b02\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") "
Oct 30 13:00:46.283010 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.282981    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d739dd4-b062-4a32-a496-637ee6476b02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d739dd4-b062-4a32-a496-637ee6476b02-istio-podinfo\") pod \"1d739dd4-b062-4a32-a496-637ee6476b02\" (UID: \"1d739dd4-b062-4a32-a496-637ee6476b02\") "
Oct 30 13:00:46.283493 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.283465    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-workload-certs" pod "1d739dd4-b062-4a32-a496-637ee6476b02" (UID: "1d739dd4-b062-4a32-a496-637ee6476b02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:46.283774 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.283750    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-istio-data" pod "1d739dd4-b062-4a32-a496-637ee6476b02" (UID: "1d739dd4-b062-4a32-a496-637ee6476b02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:46.283867 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.283786    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d739dd4-b062-4a32-a496-637ee6476b02-istiod-ca-cert" pod "1d739dd4-b062-4a32-a496-637ee6476b02" (UID: "1d739dd4-b062-4a32-a496-637ee6476b02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:46.283867 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.283845    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-credential-socket" pod "1d739dd4-b062-4a32-a496-637ee6476b02" (UID: "1d739dd4-b062-4a32-a496-637ee6476b02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:46.284061 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.284008    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-workload-socket" pod "1d739dd4-b062-4a32-a496-637ee6476b02" (UID: "1d739dd4-b062-4a32-a496-637ee6476b02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:46.284168 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.284142    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6163d41-2082-46bd-802d-38271a473e7f-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.284168 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.284165    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.284282 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.284180    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knfnm\" (UniqueName: \"kubernetes.io/projected/a6163d41-2082-46bd-802d-38271a473e7f-kube-api-access-knfnm\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.284347 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.284253    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.284444 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.284349    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.284444 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.284380    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d739dd4-b062-4a32-a496-637ee6476b02-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.284444 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.284409    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.284444 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.284422    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.284444 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.284436    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6163d41-2082-46bd-802d-38271a473e7f-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.284752 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.284448    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.284752 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.284461    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6163d41-2082-46bd-802d-38271a473e7f-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.284752 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.284474    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6163d41-2082-46bd-802d-38271a473e7f-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.285021 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.284982    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-istio-envoy" pod "1d739dd4-b062-4a32-a496-637ee6476b02" (UID: "1d739dd4-b062-4a32-a496-637ee6476b02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:46.285487 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.285455    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d739dd4-b062-4a32-a496-637ee6476b02-istio-token" pod "1d739dd4-b062-4a32-a496-637ee6476b02" (UID: "1d739dd4-b062-4a32-a496-637ee6476b02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:46.286492 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.286459    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d739dd4-b062-4a32-a496-637ee6476b02-kube-api-access-jvcmg" pod "1d739dd4-b062-4a32-a496-637ee6476b02" (UID: "1d739dd4-b062-4a32-a496-637ee6476b02"). InnerVolumeSpecName "kube-api-access-jvcmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:46.286676 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.286656    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d739dd4-b062-4a32-a496-637ee6476b02-istio-podinfo" pod "1d739dd4-b062-4a32-a496-637ee6476b02" (UID: "1d739dd4-b062-4a32-a496-637ee6476b02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:00:46.422744 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.422323    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.422744 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.422358    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvcmg\" (UniqueName: \"kubernetes.io/projected/1d739dd4-b062-4a32-a496-637ee6476b02-kube-api-access-jvcmg\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.422744 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.422374    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d739dd4-b062-4a32-a496-637ee6476b02-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.422744 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.422383    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d739dd4-b062-4a32-a496-637ee6476b02-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.422744 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.422415    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d739dd4-b062-4a32-a496-637ee6476b02-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.506774 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.506735    5837 generic.go:356] "Generic (PLEG): container finished" podID="a6163d41-2082-46bd-802d-38271a473e7f" containerID="67800d8309f8c4a7bb5120ebe83730e14ac2aae038bee7c02ef90221643da05c" exitCode=0
Oct 30 13:00:46.506971 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.506798    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"
Oct 30 13:00:46.506971 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.506839    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr" event={"ID":"a6163d41-2082-46bd-802d-38271a473e7f","Type":"ContainerDied","Data":"67800d8309f8c4a7bb5120ebe83730e14ac2aae038bee7c02ef90221643da05c"}
Oct 30 13:00:46.506971 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.506894    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr" event={"ID":"a6163d41-2082-46bd-802d-38271a473e7f","Type":"ContainerDied","Data":"943a04dd1d3366b7171c25ce66c8e4d41331a18bee3fbf6897b54b4c6126f169"}
Oct 30 13:00:46.506971 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.506938    5837 scope.go:122] "RemoveContainer" containerID="67800d8309f8c4a7bb5120ebe83730e14ac2aae038bee7c02ef90221643da05c"
Oct 30 13:00:46.510897 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.510871    5837 generic.go:356] "Generic (PLEG): container finished" podID="1d739dd4-b062-4a32-a496-637ee6476b02" containerID="971a01975f4a5a50e0206c16b3bfec554fa872d6363105324136b718fa902b10" exitCode=0
Oct 30 13:00:46.511025 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.510937    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"
Oct 30 13:00:46.511025 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.510946    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm" event={"ID":"1d739dd4-b062-4a32-a496-637ee6476b02","Type":"ContainerDied","Data":"971a01975f4a5a50e0206c16b3bfec554fa872d6363105324136b718fa902b10"}
Oct 30 13:00:46.511025 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.510980    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm" event={"ID":"1d739dd4-b062-4a32-a496-637ee6476b02","Type":"ContainerDied","Data":"7ce15977a337c20af6d3f66d963368d5dd5ae924496426691bb8c859c3ec77ec"}
Oct 30 13:00:46.522876 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.522852    5837 scope.go:122] "RemoveContainer" containerID="67800d8309f8c4a7bb5120ebe83730e14ac2aae038bee7c02ef90221643da05c"
Oct 30 13:00:46.523209 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:46.523190    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67800d8309f8c4a7bb5120ebe83730e14ac2aae038bee7c02ef90221643da05c\": container with ID starting with 67800d8309f8c4a7bb5120ebe83730e14ac2aae038bee7c02ef90221643da05c not found: ID does not exist" containerID="67800d8309f8c4a7bb5120ebe83730e14ac2aae038bee7c02ef90221643da05c"
Oct 30 13:00:46.523286 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.523219    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67800d8309f8c4a7bb5120ebe83730e14ac2aae038bee7c02ef90221643da05c"} err="failed to get container status \"67800d8309f8c4a7bb5120ebe83730e14ac2aae038bee7c02ef90221643da05c\": rpc error: code = NotFound desc = could not find container \"67800d8309f8c4a7bb5120ebe83730e14ac2aae038bee7c02ef90221643da05c\": container with ID starting with 67800d8309f8c4a7bb5120ebe83730e14ac2aae038bee7c02ef90221643da05c not found: ID does not exist"
Oct 30 13:00:46.523286 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.523237    5837 scope.go:122] "RemoveContainer" containerID="971a01975f4a5a50e0206c16b3bfec554fa872d6363105324136b718fa902b10"
Oct 30 13:00:46.534519 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.534499    5837 scope.go:122] "RemoveContainer" containerID="4a055c20d83537e538b19f936d38c372844259b21dc554847af0f977d2e61d23"
Oct 30 13:00:46.547126 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.546727    5837 scope.go:122] "RemoveContainer" containerID="971a01975f4a5a50e0206c16b3bfec554fa872d6363105324136b718fa902b10"
Oct 30 13:00:46.547331 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:46.547298    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"971a01975f4a5a50e0206c16b3bfec554fa872d6363105324136b718fa902b10\": container with ID starting with 971a01975f4a5a50e0206c16b3bfec554fa872d6363105324136b718fa902b10 not found: ID does not exist" containerID="971a01975f4a5a50e0206c16b3bfec554fa872d6363105324136b718fa902b10"
Oct 30 13:00:46.547546 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.547358    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"971a01975f4a5a50e0206c16b3bfec554fa872d6363105324136b718fa902b10"} err="failed to get container status \"971a01975f4a5a50e0206c16b3bfec554fa872d6363105324136b718fa902b10\": rpc error: code = NotFound desc = could not find container \"971a01975f4a5a50e0206c16b3bfec554fa872d6363105324136b718fa902b10\": container with ID starting with 971a01975f4a5a50e0206c16b3bfec554fa872d6363105324136b718fa902b10 not found: ID does not exist"
Oct 30 13:00:46.547546 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.547381    5837 scope.go:122] "RemoveContainer" containerID="4a055c20d83537e538b19f936d38c372844259b21dc554847af0f977d2e61d23"
Oct 30 13:00:46.547936 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:46.547915    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a055c20d83537e538b19f936d38c372844259b21dc554847af0f977d2e61d23\": container with ID starting with 4a055c20d83537e538b19f936d38c372844259b21dc554847af0f977d2e61d23 not found: ID does not exist" containerID="4a055c20d83537e538b19f936d38c372844259b21dc554847af0f977d2e61d23"
Oct 30 13:00:46.548017 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.547940    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a055c20d83537e538b19f936d38c372844259b21dc554847af0f977d2e61d23"} err="failed to get container status \"4a055c20d83537e538b19f936d38c372844259b21dc554847af0f977d2e61d23\": rpc error: code = NotFound desc = could not find container \"4a055c20d83537e538b19f936d38c372844259b21dc554847af0f977d2e61d23\": container with ID starting with 4a055c20d83537e538b19f936d38c372844259b21dc554847af0f977d2e61d23 not found: ID does not exist"
Oct 30 13:00:46.552116 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.552095    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"]
Oct 30 13:00:46.553122 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:00:46.553102    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83015f82_5f7e_4a14_9d1e_f9472c34badd.slice/crio-f15ebe3640d8a4b800e8dcddcc2f832f546be2be3f863ff6bd9aa1e8a5d86e2b WatchSource:0}: Error finding container f15ebe3640d8a4b800e8dcddcc2f832f546be2be3f863ff6bd9aa1e8a5d86e2b: Status 404 returned error can't find the container with id f15ebe3640d8a4b800e8dcddcc2f832f546be2be3f863ff6bd9aa1e8a5d86e2b
Oct 30 13:00:46.575776 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.575743    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"]
Oct 30 13:00:46.581025 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.581003    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-xcsxr"]
Oct 30 13:00:46.611725 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.611697    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"]
Oct 30 13:00:46.615421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:46.615386    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-hrbqm"]
Oct 30 13:00:47.530410 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:47.530356    5837 generic.go:356] "Generic (PLEG): container finished" podID="83015f82-5f7e-4a14-9d1e-f9472c34badd" containerID="88593a4f0aee942a8bce7fe2e6df0e5886f430f531f586a89ec6a566a3f43a47" exitCode=0
Oct 30 13:00:47.530835 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:47.530442    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz" event={"ID":"83015f82-5f7e-4a14-9d1e-f9472c34badd","Type":"ContainerDied","Data":"88593a4f0aee942a8bce7fe2e6df0e5886f430f531f586a89ec6a566a3f43a47"}
Oct 30 13:00:47.530835 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:47.530545    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz" event={"ID":"83015f82-5f7e-4a14-9d1e-f9472c34badd","Type":"ContainerStarted","Data":"f15ebe3640d8a4b800e8dcddcc2f832f546be2be3f863ff6bd9aa1e8a5d86e2b"}
Oct 30 13:00:47.606457 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:47.606425    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d739dd4-b062-4a32-a496-637ee6476b02" path="/var/lib/kubelet/pods/1d739dd4-b062-4a32-a496-637ee6476b02/volumes"
Oct 30 13:00:47.607032 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:47.607017    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6163d41-2082-46bd-802d-38271a473e7f" path="/var/lib/kubelet/pods/a6163d41-2082-46bd-802d-38271a473e7f/volumes"
Oct 30 13:00:48.715928 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:48.715902    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"
Oct 30 13:00:48.758412 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:48.758372    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83015f82-5f7e-4a14-9d1e-f9472c34badd-kube-api-access-2gn6x\" (UniqueName: \"kubernetes.io/projected/83015f82-5f7e-4a14-9d1e-f9472c34badd-kube-api-access-2gn6x\") pod \"83015f82-5f7e-4a14-9d1e-f9472c34badd\" (UID: \"83015f82-5f7e-4a14-9d1e-f9472c34badd\") "
Oct 30 13:00:48.758581 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:48.758476    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83015f82-5f7e-4a14-9d1e-f9472c34badd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83015f82-5f7e-4a14-9d1e-f9472c34badd-maas-api-openshift-ca\") pod \"83015f82-5f7e-4a14-9d1e-f9472c34badd\" (UID: \"83015f82-5f7e-4a14-9d1e-f9472c34badd\") "
Oct 30 13:00:48.758581 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:48.758527    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83015f82-5f7e-4a14-9d1e-f9472c34badd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83015f82-5f7e-4a14-9d1e-f9472c34badd-maas-api-xks-ca\") pod \"83015f82-5f7e-4a14-9d1e-f9472c34badd\" (UID: \"83015f82-5f7e-4a14-9d1e-f9472c34badd\") "
Oct 30 13:00:48.759643 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:48.759611    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83015f82-5f7e-4a14-9d1e-f9472c34badd-maas-api-openshift-ca" pod "83015f82-5f7e-4a14-9d1e-f9472c34badd" (UID: "83015f82-5f7e-4a14-9d1e-f9472c34badd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:48.761016 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:48.760991    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83015f82-5f7e-4a14-9d1e-f9472c34badd-kube-api-access-2gn6x" pod "83015f82-5f7e-4a14-9d1e-f9472c34badd" (UID: "83015f82-5f7e-4a14-9d1e-f9472c34badd"). InnerVolumeSpecName "kube-api-access-2gn6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:48.761485 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:48.761447    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83015f82-5f7e-4a14-9d1e-f9472c34badd-maas-api-xks-ca" pod "83015f82-5f7e-4a14-9d1e-f9472c34badd" (UID: "83015f82-5f7e-4a14-9d1e-f9472c34badd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:00:48.861011 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:48.860940    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gn6x\" (UniqueName: \"kubernetes.io/projected/83015f82-5f7e-4a14-9d1e-f9472c34badd-kube-api-access-2gn6x\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:48.861011 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:48.860966    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83015f82-5f7e-4a14-9d1e-f9472c34badd-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:48.861011 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:48.860977    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83015f82-5f7e-4a14-9d1e-f9472c34badd-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:49.548430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:49.548380    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz" event={"ID":"83015f82-5f7e-4a14-9d1e-f9472c34badd","Type":"ContainerDied","Data":"f15ebe3640d8a4b800e8dcddcc2f832f546be2be3f863ff6bd9aa1e8a5d86e2b"}
Oct 30 13:00:49.548622 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:49.548442    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f15ebe3640d8a4b800e8dcddcc2f832f546be2be3f863ff6bd9aa1e8a5d86e2b"
Oct 30 13:00:49.548622 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:49.548451    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"
Oct 30 13:00:50.965489 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:50.965453    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"]
Oct 30 13:00:50.998332 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:50.998305    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"]
Oct 30 13:00:55.966532 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:55.966465    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz" podUID="9abf7013-9e1f-475c-9fb9-9e928a7dcb79" containerName="payload-processing" containerID="cri-o://e205a7a0b86b8ea174fc73571c2ab4d94b5cf43aa6991bd4bbfea99656cdc085" gracePeriod=25
Oct 30 13:00:55.998744 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:55.998712    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk" podUID="53dd4fc3-081f-4b41-bedc-d6b87a44aec3" containerName="payload-pre-processing" containerID="cri-o://a1b13d18d66972a525913da81918d8efc4f0ea88aa5e9737b632e921f882e4c7" gracePeriod=25
Oct 30 13:00:56.339601 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.339574    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"
Oct 30 13:00:56.384333 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.384304    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"
Oct 30 13:00:56.526386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.526316    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9abf7013-9e1f-475c-9fb9-9e928a7dcb79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abf7013-9e1f-475c-9fb9-9e928a7dcb79-plugins-config-volume\") pod \"9abf7013-9e1f-475c-9fb9-9e928a7dcb79\" (UID: \"9abf7013-9e1f-475c-9fb9-9e928a7dcb79\") "
Oct 30 13:00:56.526646 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.526624    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abf7013-9e1f-475c-9fb9-9e928a7dcb79-kube-api-access-7rjk8\" (UniqueName: \"kubernetes.io/projected/9abf7013-9e1f-475c-9fb9-9e928a7dcb79-kube-api-access-7rjk8\") pod \"9abf7013-9e1f-475c-9fb9-9e928a7dcb79\" (UID: \"9abf7013-9e1f-475c-9fb9-9e928a7dcb79\") "
Oct 30 13:00:56.526745 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.526729    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53dd4fc3-081f-4b41-bedc-d6b87a44aec3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53dd4fc3-081f-4b41-bedc-d6b87a44aec3-plugins-config-volume\") pod \"53dd4fc3-081f-4b41-bedc-d6b87a44aec3\" (UID: \"53dd4fc3-081f-4b41-bedc-d6b87a44aec3\") "
Oct 30 13:00:56.526781 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.526746    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abf7013-9e1f-475c-9fb9-9e928a7dcb79-plugins-config-volume" pod "9abf7013-9e1f-475c-9fb9-9e928a7dcb79" (UID: "9abf7013-9e1f-475c-9fb9-9e928a7dcb79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:56.526834 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.526807    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53dd4fc3-081f-4b41-bedc-d6b87a44aec3-kube-api-access-x4h8v\" (UniqueName: \"kubernetes.io/projected/53dd4fc3-081f-4b41-bedc-d6b87a44aec3-kube-api-access-x4h8v\") pod \"53dd4fc3-081f-4b41-bedc-d6b87a44aec3\" (UID: \"53dd4fc3-081f-4b41-bedc-d6b87a44aec3\") "
Oct 30 13:00:56.527097 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.527070    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53dd4fc3-081f-4b41-bedc-d6b87a44aec3-plugins-config-volume" pod "53dd4fc3-081f-4b41-bedc-d6b87a44aec3" (UID: "53dd4fc3-081f-4b41-bedc-d6b87a44aec3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:56.527478 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.527460    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53dd4fc3-081f-4b41-bedc-d6b87a44aec3-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:56.527536 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.527480    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abf7013-9e1f-475c-9fb9-9e928a7dcb79-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:56.528943 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.528917    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abf7013-9e1f-475c-9fb9-9e928a7dcb79-kube-api-access-7rjk8" pod "9abf7013-9e1f-475c-9fb9-9e928a7dcb79" (UID: "9abf7013-9e1f-475c-9fb9-9e928a7dcb79"). InnerVolumeSpecName "kube-api-access-7rjk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:56.528943 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.528935    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53dd4fc3-081f-4b41-bedc-d6b87a44aec3-kube-api-access-x4h8v" pod "53dd4fc3-081f-4b41-bedc-d6b87a44aec3" (UID: "53dd4fc3-081f-4b41-bedc-d6b87a44aec3"). InnerVolumeSpecName "kube-api-access-x4h8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:56.620095 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.620060    5837 generic.go:356] "Generic (PLEG): container finished" podID="9abf7013-9e1f-475c-9fb9-9e928a7dcb79" containerID="e205a7a0b86b8ea174fc73571c2ab4d94b5cf43aa6991bd4bbfea99656cdc085" exitCode=0
Oct 30 13:00:56.620273 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.620117    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"
Oct 30 13:00:56.620273 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.620153    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz" event={"ID":"9abf7013-9e1f-475c-9fb9-9e928a7dcb79","Type":"ContainerDied","Data":"e205a7a0b86b8ea174fc73571c2ab4d94b5cf43aa6991bd4bbfea99656cdc085"}
Oct 30 13:00:56.620273 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.620223    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz" event={"ID":"9abf7013-9e1f-475c-9fb9-9e928a7dcb79","Type":"ContainerDied","Data":"76aa59d4ef26d9cb3296065aafd55c0f0d113bfbdd17b6a84bed36e61e237cd0"}
Oct 30 13:00:56.620273 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.620244    5837 scope.go:122] "RemoveContainer" containerID="e205a7a0b86b8ea174fc73571c2ab4d94b5cf43aa6991bd4bbfea99656cdc085"
Oct 30 13:00:56.621686 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.621659    5837 generic.go:356] "Generic (PLEG): container finished" podID="53dd4fc3-081f-4b41-bedc-d6b87a44aec3" containerID="a1b13d18d66972a525913da81918d8efc4f0ea88aa5e9737b632e921f882e4c7" exitCode=0
Oct 30 13:00:56.621794 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.621722    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"
Oct 30 13:00:56.621794 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.621754    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk" event={"ID":"53dd4fc3-081f-4b41-bedc-d6b87a44aec3","Type":"ContainerDied","Data":"a1b13d18d66972a525913da81918d8efc4f0ea88aa5e9737b632e921f882e4c7"}
Oct 30 13:00:56.621894 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.621797    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk" event={"ID":"53dd4fc3-081f-4b41-bedc-d6b87a44aec3","Type":"ContainerDied","Data":"633cb6060bbe0f6a844979aa37d4b4fc55a9c970925db9e31b5a4601f4a6e271"}
Oct 30 13:00:56.628354 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.628335    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4h8v\" (UniqueName: \"kubernetes.io/projected/53dd4fc3-081f-4b41-bedc-d6b87a44aec3-kube-api-access-x4h8v\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:56.628448 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.628356    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rjk8\" (UniqueName: \"kubernetes.io/projected/9abf7013-9e1f-475c-9fb9-9e928a7dcb79-kube-api-access-7rjk8\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:00:56.632883 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.632857    5837 scope.go:122] "RemoveContainer" containerID="e205a7a0b86b8ea174fc73571c2ab4d94b5cf43aa6991bd4bbfea99656cdc085"
Oct 30 13:00:56.633254 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:56.633233    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e205a7a0b86b8ea174fc73571c2ab4d94b5cf43aa6991bd4bbfea99656cdc085\": container with ID starting with e205a7a0b86b8ea174fc73571c2ab4d94b5cf43aa6991bd4bbfea99656cdc085 not found: ID does not exist" containerID="e205a7a0b86b8ea174fc73571c2ab4d94b5cf43aa6991bd4bbfea99656cdc085"
Oct 30 13:00:56.633347 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.633264    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e205a7a0b86b8ea174fc73571c2ab4d94b5cf43aa6991bd4bbfea99656cdc085"} err="failed to get container status \"e205a7a0b86b8ea174fc73571c2ab4d94b5cf43aa6991bd4bbfea99656cdc085\": rpc error: code = NotFound desc = could not find container \"e205a7a0b86b8ea174fc73571c2ab4d94b5cf43aa6991bd4bbfea99656cdc085\": container with ID starting with e205a7a0b86b8ea174fc73571c2ab4d94b5cf43aa6991bd4bbfea99656cdc085 not found: ID does not exist"
Oct 30 13:00:56.633347 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.633286    5837 scope.go:122] "RemoveContainer" containerID="a1b13d18d66972a525913da81918d8efc4f0ea88aa5e9737b632e921f882e4c7"
Oct 30 13:00:56.645362 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.645338    5837 scope.go:122] "RemoveContainer" containerID="a1b13d18d66972a525913da81918d8efc4f0ea88aa5e9737b632e921f882e4c7"
Oct 30 13:00:56.645807 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:00:56.645788    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b13d18d66972a525913da81918d8efc4f0ea88aa5e9737b632e921f882e4c7\": container with ID starting with a1b13d18d66972a525913da81918d8efc4f0ea88aa5e9737b632e921f882e4c7 not found: ID does not exist" containerID="a1b13d18d66972a525913da81918d8efc4f0ea88aa5e9737b632e921f882e4c7"
Oct 30 13:00:56.645863 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.645815    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b13d18d66972a525913da81918d8efc4f0ea88aa5e9737b632e921f882e4c7"} err="failed to get container status \"a1b13d18d66972a525913da81918d8efc4f0ea88aa5e9737b632e921f882e4c7\": rpc error: code = NotFound desc = could not find container \"a1b13d18d66972a525913da81918d8efc4f0ea88aa5e9737b632e921f882e4c7\": container with ID starting with a1b13d18d66972a525913da81918d8efc4f0ea88aa5e9737b632e921f882e4c7 not found: ID does not exist"
Oct 30 13:00:56.671153 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.671102    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"]
Oct 30 13:00:56.674942 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.674920    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-56c74bd459-jtrmk"]
Oct 30 13:00:56.697708 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.697683    5837 scope.go:122] "RemoveContainer" containerID="267f955ce4c8140fb04af0dbd345193c3d0ec40d4875e8533370cd41ceca82c6"
Oct 30 13:00:56.708813 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.708537    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"]
Oct 30 13:00:56.708813 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.708545    5837 scope.go:122] "RemoveContainer" containerID="657f28bd59b471d3c024f4e30d6d96afb870f5d6296ca998c1c2fea388d94890"
Oct 30 13:00:56.711815 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.711792    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096cc3-6b7597fb4b-sjwqz"]
Oct 30 13:00:56.718670 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:56.718650    5837 scope.go:122] "RemoveContainer" containerID="d2995074f1b84bca3027258868bef0165534dd4c4f8e22a2536acade2bbeb3ac"
Oct 30 13:00:57.606830 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:57.606801    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53dd4fc3-081f-4b41-bedc-d6b87a44aec3" path="/var/lib/kubelet/pods/53dd4fc3-081f-4b41-bedc-d6b87a44aec3/volumes"
Oct 30 13:00:57.607253 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:57.607238    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9abf7013-9e1f-475c-9fb9-9e928a7dcb79" path="/var/lib/kubelet/pods/9abf7013-9e1f-475c-9fb9-9e928a7dcb79/volumes"
Oct 30 13:00:59.613355 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.613323    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"]
Oct 30 13:00:59.614086 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614070    5837 cpu_manager.go:414] "containerMap: removing container" podUID="1d739dd4-b062-4a32-a496-637ee6476b02" containerName="istio-proxy"
Oct 30 13:00:59.614139 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614088    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d739dd4-b062-4a32-a496-637ee6476b02" containerName="istio-proxy"
Oct 30 13:00:59.614139 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614100    5837 cpu_manager.go:414] "containerMap: removing container" podUID="53dd4fc3-081f-4b41-bedc-d6b87a44aec3" containerName="payload-pre-processing"
Oct 30 13:00:59.614139 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614106    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53dd4fc3-081f-4b41-bedc-d6b87a44aec3" containerName="payload-pre-processing"
Oct 30 13:00:59.614139 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614129    5837 cpu_manager.go:414] "containerMap: removing container" podUID="9abf7013-9e1f-475c-9fb9-9e928a7dcb79" containerName="payload-processing"
Oct 30 13:00:59.614139 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614136    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abf7013-9e1f-475c-9fb9-9e928a7dcb79" containerName="payload-processing"
Oct 30 13:00:59.614296 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614151    5837 cpu_manager.go:414] "containerMap: removing container" podUID="a6163d41-2082-46bd-802d-38271a473e7f" containerName="istio-proxy"
Oct 30 13:00:59.614296 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614156    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6163d41-2082-46bd-802d-38271a473e7f" containerName="istio-proxy"
Oct 30 13:00:59.614296 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614186    5837 cpu_manager.go:414] "containerMap: removing container" podUID="83015f82-5f7e-4a14-9d1e-f9472c34badd" containerName="revoke-keys"
Oct 30 13:00:59.614296 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614190    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83015f82-5f7e-4a14-9d1e-f9472c34badd" containerName="revoke-keys"
Oct 30 13:00:59.614296 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614296    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d739dd4-b062-4a32-a496-637ee6476b02" containerName="istio-proxy"
Oct 30 13:00:59.614461 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614304    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="53dd4fc3-081f-4b41-bedc-d6b87a44aec3" containerName="payload-pre-processing"
Oct 30 13:00:59.614461 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614312    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abf7013-9e1f-475c-9fb9-9e928a7dcb79" containerName="payload-processing"
Oct 30 13:00:59.614461 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614319    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6163d41-2082-46bd-802d-38271a473e7f" containerName="istio-proxy"
Oct 30 13:00:59.614461 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614328    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d739dd4-b062-4a32-a496-637ee6476b02" containerName="istio-proxy"
Oct 30 13:00:59.614461 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.614334    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="83015f82-5f7e-4a14-9d1e-f9472c34badd" containerName="revoke-keys"
Oct 30 13:00:59.619435 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.619369    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"
Oct 30 13:00:59.639297 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.639261    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"]
Oct 30 13:00:59.649856 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.646738    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-026c2f-dockercfg-bkrb8\""
Oct 30 13:00:59.649856 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.646949    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-026c2f\""
Oct 30 13:00:59.787987 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.783482    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d126169-6889-474a-a6e9-bda03dc1df4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd\" (UID: \"1d126169-6889-474a-a6e9-bda03dc1df4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"
Oct 30 13:00:59.787987 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.783671    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rst7c\" (UniqueName: \"kubernetes.io/projected/1d126169-6889-474a-a6e9-bda03dc1df4c-kube-api-access-rst7c\") pod \"payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd\" (UID: \"1d126169-6889-474a-a6e9-bda03dc1df4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"
Oct 30 13:00:59.787987 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.787870    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"]
Oct 30 13:00:59.788896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.788869    5837 cpu_manager.go:414] "containerMap: removing container" podUID="1d739dd4-b062-4a32-a496-637ee6476b02" containerName="istio-proxy"
Oct 30 13:00:59.788896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.788894    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d739dd4-b062-4a32-a496-637ee6476b02" containerName="istio-proxy"
Oct 30 13:00:59.803347 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.803314    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"]
Oct 30 13:00:59.803581 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.803445    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"
Oct 30 13:00:59.885110 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.885018    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d126169-6889-474a-a6e9-bda03dc1df4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd\" (UID: \"1d126169-6889-474a-a6e9-bda03dc1df4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"
Oct 30 13:00:59.885268 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.885136    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8xls\" (UniqueName: \"kubernetes.io/projected/5ae1fb09-7412-451d-b4d9-966a08372805-kube-api-access-p8xls\") pod \"payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56\" (UID: \"5ae1fb09-7412-451d-b4d9-966a08372805\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"
Oct 30 13:00:59.885268 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.885183    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rst7c\" (UniqueName: \"kubernetes.io/projected/1d126169-6889-474a-a6e9-bda03dc1df4c-kube-api-access-rst7c\") pod \"payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd\" (UID: \"1d126169-6889-474a-a6e9-bda03dc1df4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"
Oct 30 13:00:59.885268 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.885202    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae1fb09-7412-451d-b4d9-966a08372805-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56\" (UID: \"5ae1fb09-7412-451d-b4d9-966a08372805\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"
Oct 30 13:00:59.885829 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.885806    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d126169-6889-474a-a6e9-bda03dc1df4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd\" (UID: \"1d126169-6889-474a-a6e9-bda03dc1df4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"
Oct 30 13:00:59.904261 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.904233    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rst7c\" (UniqueName: \"kubernetes.io/projected/1d126169-6889-474a-a6e9-bda03dc1df4c-kube-api-access-rst7c\") pod \"payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd\" (UID: \"1d126169-6889-474a-a6e9-bda03dc1df4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"
Oct 30 13:00:59.938152 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.938119    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"
Oct 30 13:00:59.987083 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.987037    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae1fb09-7412-451d-b4d9-966a08372805-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56\" (UID: \"5ae1fb09-7412-451d-b4d9-966a08372805\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"
Oct 30 13:00:59.988180 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.988128    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8xls\" (UniqueName: \"kubernetes.io/projected/5ae1fb09-7412-451d-b4d9-966a08372805-kube-api-access-p8xls\") pod \"payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56\" (UID: \"5ae1fb09-7412-451d-b4d9-966a08372805\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"
Oct 30 13:00:59.988812 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:00:59.988790    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae1fb09-7412-451d-b4d9-966a08372805-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56\" (UID: \"5ae1fb09-7412-451d-b4d9-966a08372805\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"
Oct 30 13:01:00.007147 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:00.007113    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8xls\" (UniqueName: \"kubernetes.io/projected/5ae1fb09-7412-451d-b4d9-966a08372805-kube-api-access-p8xls\") pod \"payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56\" (UID: \"5ae1fb09-7412-451d-b4d9-966a08372805\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"
Oct 30 13:01:00.137015 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:00.136936    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"
Oct 30 13:01:00.247288 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:00.247259    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"]
Oct 30 13:01:00.248516 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:01:00.248411    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d126169_6889_474a_a6e9_bda03dc1df4c.slice/crio-f0596c67a1a2113efdce5e194af77369fe495713da14f71810113d00f049dfbd WatchSource:0}: Error finding container f0596c67a1a2113efdce5e194af77369fe495713da14f71810113d00f049dfbd: Status 404 returned error can't find the container with id f0596c67a1a2113efdce5e194af77369fe495713da14f71810113d00f049dfbd
Oct 30 13:01:00.454733 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:00.454192    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"]
Oct 30 13:01:00.456921 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:01:00.456893    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae1fb09_7412_451d_b4d9_966a08372805.slice/crio-020286d0f7650ba2152f041141523ac0294c951aaa0a282b675cc7a93718d56f WatchSource:0}: Error finding container 020286d0f7650ba2152f041141523ac0294c951aaa0a282b675cc7a93718d56f: Status 404 returned error can't find the container with id 020286d0f7650ba2152f041141523ac0294c951aaa0a282b675cc7a93718d56f
Oct 30 13:01:00.658591 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:00.658471    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56" event={"ID":"5ae1fb09-7412-451d-b4d9-966a08372805","Type":"ContainerStarted","Data":"c864f3da0f5f426398aeb678cd9891e5150ea18505ae4904324a4db0236c207b"}
Oct 30 13:01:00.658591 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:00.658535    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"
Oct 30 13:01:00.658591 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:00.658558    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56" event={"ID":"5ae1fb09-7412-451d-b4d9-966a08372805","Type":"ContainerStarted","Data":"020286d0f7650ba2152f041141523ac0294c951aaa0a282b675cc7a93718d56f"}
Oct 30 13:01:00.660578 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:00.660550    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd" event={"ID":"1d126169-6889-474a-a6e9-bda03dc1df4c","Type":"ContainerStarted","Data":"ab50ec505bdca3e730fdc315557ca9d944c949aca8991e0668819d330a37eb27"}
Oct 30 13:01:00.660697 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:00.660604    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd" event={"ID":"1d126169-6889-474a-a6e9-bda03dc1df4c","Type":"ContainerStarted","Data":"f0596c67a1a2113efdce5e194af77369fe495713da14f71810113d00f049dfbd"}
Oct 30 13:01:00.660697 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:00.660642    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"
Oct 30 13:01:00.699516 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:00.699032    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56" podStartSLOduration=1.699014521 podStartE2EDuration="1.699014521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:00.69588879 +0000 UTC m=+1449.220173010" watchObservedRunningTime="2026-10-30 13:01:00.699014521 +0000 UTC m=+1449.223298742"
Oct 30 13:01:00.734992 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:00.734948    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd" podStartSLOduration=1.734930114 podStartE2EDuration="1.734930114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:00.732605852 +0000 UTC m=+1449.256890076" watchObservedRunningTime="2026-10-30 13:01:00.734930114 +0000 UTC m=+1449.259214338"
Oct 30 13:01:01.892528 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:01.892451    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"]
Oct 30 13:01:01.892913 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:01.892680    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p" podUID="6e913224-538a-4f4a-ba4f-c87d3446cb96" containerName="payload-processing" containerID="cri-o://e8449940fcdc1258abffa00e7c651103d5c5b8928a1e70d4bff53c7ce804fc86" gracePeriod=30
Oct 30 13:01:01.909726 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:01.909692    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"]
Oct 30 13:01:01.909977 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:01.909950    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff" podUID="ed765745-1281-4261-a959-0695a04713b1" containerName="payload-pre-processing" containerID="cri-o://68279432dc3cab35abe8114994ce01db0a1e157923caa8a771d70eaa5ffdcee6" gracePeriod=30
Oct 30 13:01:02.314451 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.310647    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"
Oct 30 13:01:02.321284 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.317625    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"
Oct 30 13:01:02.451505 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.440962    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e913224-538a-4f4a-ba4f-c87d3446cb96-kube-api-access-ntk4p\" (UniqueName: \"kubernetes.io/projected/6e913224-538a-4f4a-ba4f-c87d3446cb96-kube-api-access-ntk4p\") pod \"6e913224-538a-4f4a-ba4f-c87d3446cb96\" (UID: \"6e913224-538a-4f4a-ba4f-c87d3446cb96\") "
Oct 30 13:01:02.451505 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.441158    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e913224-538a-4f4a-ba4f-c87d3446cb96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e913224-538a-4f4a-ba4f-c87d3446cb96-plugins-config-volume\") pod \"6e913224-538a-4f4a-ba4f-c87d3446cb96\" (UID: \"6e913224-538a-4f4a-ba4f-c87d3446cb96\") "
Oct 30 13:01:02.451505 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.441354    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed765745-1281-4261-a959-0695a04713b1-kube-api-access-mq59x\" (UniqueName: \"kubernetes.io/projected/ed765745-1281-4261-a959-0695a04713b1-kube-api-access-mq59x\") pod \"ed765745-1281-4261-a959-0695a04713b1\" (UID: \"ed765745-1281-4261-a959-0695a04713b1\") "
Oct 30 13:01:02.451505 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.441476    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed765745-1281-4261-a959-0695a04713b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed765745-1281-4261-a959-0695a04713b1-plugins-config-volume\") pod \"ed765745-1281-4261-a959-0695a04713b1\" (UID: \"ed765745-1281-4261-a959-0695a04713b1\") "
Oct 30 13:01:02.451505 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.442554    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed765745-1281-4261-a959-0695a04713b1-plugins-config-volume" pod "ed765745-1281-4261-a959-0695a04713b1" (UID: "ed765745-1281-4261-a959-0695a04713b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:02.451505 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.443530    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e913224-538a-4f4a-ba4f-c87d3446cb96-plugins-config-volume" pod "6e913224-538a-4f4a-ba4f-c87d3446cb96" (UID: "6e913224-538a-4f4a-ba4f-c87d3446cb96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:02.453888 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.453843    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e913224-538a-4f4a-ba4f-c87d3446cb96-kube-api-access-ntk4p" pod "6e913224-538a-4f4a-ba4f-c87d3446cb96" (UID: "6e913224-538a-4f4a-ba4f-c87d3446cb96"). InnerVolumeSpecName "kube-api-access-ntk4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:02.475638 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.462630    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed765745-1281-4261-a959-0695a04713b1-kube-api-access-mq59x" pod "ed765745-1281-4261-a959-0695a04713b1" (UID: "ed765745-1281-4261-a959-0695a04713b1"). InnerVolumeSpecName "kube-api-access-mq59x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:02.544429 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.544368    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e913224-538a-4f4a-ba4f-c87d3446cb96-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:02.544628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.544436    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq59x\" (UniqueName: \"kubernetes.io/projected/ed765745-1281-4261-a959-0695a04713b1-kube-api-access-mq59x\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:02.544628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.544459    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed765745-1281-4261-a959-0695a04713b1-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:02.544628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.544473    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntk4p\" (UniqueName: \"kubernetes.io/projected/6e913224-538a-4f4a-ba4f-c87d3446cb96-kube-api-access-ntk4p\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:02.651794 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.651132    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"]
Oct 30 13:01:02.656212 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.656094    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0a01c-6v2rz"]
Oct 30 13:01:02.676102 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.676071    5837 generic.go:356] "Generic (PLEG): container finished" podID="ed765745-1281-4261-a959-0695a04713b1" containerID="68279432dc3cab35abe8114994ce01db0a1e157923caa8a771d70eaa5ffdcee6" exitCode=0
Oct 30 13:01:02.676275 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.676116    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"
Oct 30 13:01:02.676275 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.676154    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff" event={"ID":"ed765745-1281-4261-a959-0695a04713b1","Type":"ContainerDied","Data":"68279432dc3cab35abe8114994ce01db0a1e157923caa8a771d70eaa5ffdcee6"}
Oct 30 13:01:02.676275 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.676203    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff" event={"ID":"ed765745-1281-4261-a959-0695a04713b1","Type":"ContainerDied","Data":"b37a490144a41b9c112f30f8dcb01cfd9d9663e615d3b5fd759f4ed668d643f4"}
Oct 30 13:01:02.676275 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.676226    5837 scope.go:122] "RemoveContainer" containerID="68279432dc3cab35abe8114994ce01db0a1e157923caa8a771d70eaa5ffdcee6"
Oct 30 13:01:02.677793 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.677770    5837 generic.go:356] "Generic (PLEG): container finished" podID="6e913224-538a-4f4a-ba4f-c87d3446cb96" containerID="e8449940fcdc1258abffa00e7c651103d5c5b8928a1e70d4bff53c7ce804fc86" exitCode=0
Oct 30 13:01:02.677926 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.677804    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p" event={"ID":"6e913224-538a-4f4a-ba4f-c87d3446cb96","Type":"ContainerDied","Data":"e8449940fcdc1258abffa00e7c651103d5c5b8928a1e70d4bff53c7ce804fc86"}
Oct 30 13:01:02.677926 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.677843    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p" event={"ID":"6e913224-538a-4f4a-ba4f-c87d3446cb96","Type":"ContainerDied","Data":"b2ab1a277906483bd046a39b8d70095aa6ed2101e94516818e949e2e12274593"}
Oct 30 13:01:02.677926 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.677808    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"
Oct 30 13:01:02.712292 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.712258    5837 scope.go:122] "RemoveContainer" containerID="68279432dc3cab35abe8114994ce01db0a1e157923caa8a771d70eaa5ffdcee6"
Oct 30 13:01:02.715777 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:01:02.713521    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68279432dc3cab35abe8114994ce01db0a1e157923caa8a771d70eaa5ffdcee6\": container with ID starting with 68279432dc3cab35abe8114994ce01db0a1e157923caa8a771d70eaa5ffdcee6 not found: ID does not exist" containerID="68279432dc3cab35abe8114994ce01db0a1e157923caa8a771d70eaa5ffdcee6"
Oct 30 13:01:02.715777 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.713556    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68279432dc3cab35abe8114994ce01db0a1e157923caa8a771d70eaa5ffdcee6"} err="failed to get container status \"68279432dc3cab35abe8114994ce01db0a1e157923caa8a771d70eaa5ffdcee6\": rpc error: code = NotFound desc = could not find container \"68279432dc3cab35abe8114994ce01db0a1e157923caa8a771d70eaa5ffdcee6\": container with ID starting with 68279432dc3cab35abe8114994ce01db0a1e157923caa8a771d70eaa5ffdcee6 not found: ID does not exist"
Oct 30 13:01:02.715777 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.713582    5837 scope.go:122] "RemoveContainer" containerID="e8449940fcdc1258abffa00e7c651103d5c5b8928a1e70d4bff53c7ce804fc86"
Oct 30 13:01:02.748357 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.748320    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"]
Oct 30 13:01:02.749589 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.749564    5837 scope.go:122] "RemoveContainer" containerID="e8449940fcdc1258abffa00e7c651103d5c5b8928a1e70d4bff53c7ce804fc86"
Oct 30 13:01:02.750036 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:01:02.750007    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8449940fcdc1258abffa00e7c651103d5c5b8928a1e70d4bff53c7ce804fc86\": container with ID starting with e8449940fcdc1258abffa00e7c651103d5c5b8928a1e70d4bff53c7ce804fc86 not found: ID does not exist" containerID="e8449940fcdc1258abffa00e7c651103d5c5b8928a1e70d4bff53c7ce804fc86"
Oct 30 13:01:02.750145 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.750049    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8449940fcdc1258abffa00e7c651103d5c5b8928a1e70d4bff53c7ce804fc86"} err="failed to get container status \"e8449940fcdc1258abffa00e7c651103d5c5b8928a1e70d4bff53c7ce804fc86\": rpc error: code = NotFound desc = could not find container \"e8449940fcdc1258abffa00e7c651103d5c5b8928a1e70d4bff53c7ce804fc86\": container with ID starting with e8449940fcdc1258abffa00e7c651103d5c5b8928a1e70d4bff53c7ce804fc86 not found: ID does not exist"
Oct 30 13:01:02.753356 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.753329    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-548b56cf9d-czh7p"]
Oct 30 13:01:02.784327 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.784294    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"]
Oct 30 13:01:02.787661 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:02.787638    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0a01c-64db95489f-htnff"]
Oct 30 13:01:03.606747 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:03.606719    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e913224-538a-4f4a-ba4f-c87d3446cb96" path="/var/lib/kubelet/pods/6e913224-538a-4f4a-ba4f-c87d3446cb96/volumes"
Oct 30 13:01:03.607233 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:03.607218    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83015f82-5f7e-4a14-9d1e-f9472c34badd" path="/var/lib/kubelet/pods/83015f82-5f7e-4a14-9d1e-f9472c34badd/volumes"
Oct 30 13:01:03.607687 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:03.607672    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed765745-1281-4261-a959-0695a04713b1" path="/var/lib/kubelet/pods/ed765745-1281-4261-a959-0695a04713b1/volumes"
Oct 30 13:01:10.186422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.178461    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"]
Oct 30 13:01:10.186422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.179573    5837 cpu_manager.go:414] "containerMap: removing container" podUID="ed765745-1281-4261-a959-0695a04713b1" containerName="payload-pre-processing"
Oct 30 13:01:10.186422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.179594    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed765745-1281-4261-a959-0695a04713b1" containerName="payload-pre-processing"
Oct 30 13:01:10.186422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.179620    5837 cpu_manager.go:414] "containerMap: removing container" podUID="6e913224-538a-4f4a-ba4f-c87d3446cb96" containerName="payload-processing"
Oct 30 13:01:10.186422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.179630    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e913224-538a-4f4a-ba4f-c87d3446cb96" containerName="payload-processing"
Oct 30 13:01:10.186422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.179832    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed765745-1281-4261-a959-0695a04713b1" containerName="payload-pre-processing"
Oct 30 13:01:10.186422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.179901    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e913224-538a-4f4a-ba4f-c87d3446cb96" containerName="payload-processing"
Oct 30 13:01:10.217418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.214649    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"]
Oct 30 13:01:10.217418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.214771    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"
Oct 30 13:01:10.252703 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.252672    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-096cc3-dockercfg-r96w5\""
Oct 30 13:01:10.252912 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.252755    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-096cc3\""
Oct 30 13:01:10.313387 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.313346    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhbzh\" (UniqueName: \"kubernetes.io/projected/4d021c3e-5871-4c5a-aa65-0244db8501de-kube-api-access-bhbzh\") pod \"payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2\" (UID: \"4d021c3e-5871-4c5a-aa65-0244db8501de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"
Oct 30 13:01:10.313645 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.313625    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d021c3e-5871-4c5a-aa65-0244db8501de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2\" (UID: \"4d021c3e-5871-4c5a-aa65-0244db8501de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"
Oct 30 13:01:10.325849 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.325813    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"]
Oct 30 13:01:10.332013 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.331986    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"
Oct 30 13:01:10.350831 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.350783    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"]
Oct 30 13:01:10.414797 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.414760    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sck54\" (UniqueName: \"kubernetes.io/projected/d17210fa-f13e-4df6-b6c6-f2f6cbc338e3-kube-api-access-sck54\") pod \"payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp\" (UID: \"d17210fa-f13e-4df6-b6c6-f2f6cbc338e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"
Oct 30 13:01:10.414974 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.414881    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhbzh\" (UniqueName: \"kubernetes.io/projected/4d021c3e-5871-4c5a-aa65-0244db8501de-kube-api-access-bhbzh\") pod \"payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2\" (UID: \"4d021c3e-5871-4c5a-aa65-0244db8501de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"
Oct 30 13:01:10.415028 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.414982    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17210fa-f13e-4df6-b6c6-f2f6cbc338e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp\" (UID: \"d17210fa-f13e-4df6-b6c6-f2f6cbc338e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"
Oct 30 13:01:10.415136 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.415112    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d021c3e-5871-4c5a-aa65-0244db8501de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2\" (UID: \"4d021c3e-5871-4c5a-aa65-0244db8501de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"
Oct 30 13:01:10.415903 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.415886    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d021c3e-5871-4c5a-aa65-0244db8501de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2\" (UID: \"4d021c3e-5871-4c5a-aa65-0244db8501de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"
Oct 30 13:01:10.434461 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.434435    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhbzh\" (UniqueName: \"kubernetes.io/projected/4d021c3e-5871-4c5a-aa65-0244db8501de-kube-api-access-bhbzh\") pod \"payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2\" (UID: \"4d021c3e-5871-4c5a-aa65-0244db8501de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"
Oct 30 13:01:10.516529 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.516501    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sck54\" (UniqueName: \"kubernetes.io/projected/d17210fa-f13e-4df6-b6c6-f2f6cbc338e3-kube-api-access-sck54\") pod \"payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp\" (UID: \"d17210fa-f13e-4df6-b6c6-f2f6cbc338e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"
Oct 30 13:01:10.516708 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.516622    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17210fa-f13e-4df6-b6c6-f2f6cbc338e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp\" (UID: \"d17210fa-f13e-4df6-b6c6-f2f6cbc338e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"
Oct 30 13:01:10.517298 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.517272    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17210fa-f13e-4df6-b6c6-f2f6cbc338e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp\" (UID: \"d17210fa-f13e-4df6-b6c6-f2f6cbc338e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"
Oct 30 13:01:10.535408 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.535369    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sck54\" (UniqueName: \"kubernetes.io/projected/d17210fa-f13e-4df6-b6c6-f2f6cbc338e3-kube-api-access-sck54\") pod \"payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp\" (UID: \"d17210fa-f13e-4df6-b6c6-f2f6cbc338e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"
Oct 30 13:01:10.547428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.547403    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"
Oct 30 13:01:10.659580 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.659549    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"
Oct 30 13:01:10.847149 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.847120    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"]
Oct 30 13:01:10.847264 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:01:10.847249    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d021c3e_5871_4c5a_aa65_0244db8501de.slice/crio-95d17bb336675c070d104e0d714bfff92ddcc53fcaa128a5c27e41954271c5f5 WatchSource:0}: Error finding container 95d17bb336675c070d104e0d714bfff92ddcc53fcaa128a5c27e41954271c5f5: Status 404 returned error can't find the container with id 95d17bb336675c070d104e0d714bfff92ddcc53fcaa128a5c27e41954271c5f5
Oct 30 13:01:10.959801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:10.959776    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"]
Oct 30 13:01:10.960025 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:01:10.960006    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17210fa_f13e_4df6_b6c6_f2f6cbc338e3.slice/crio-283f295628543d695d8b58c8cca41ffc9a5e5714c3708cf6960a749363a8f995 WatchSource:0}: Error finding container 283f295628543d695d8b58c8cca41ffc9a5e5714c3708cf6960a749363a8f995: Status 404 returned error can't find the container with id 283f295628543d695d8b58c8cca41ffc9a5e5714c3708cf6960a749363a8f995
Oct 30 13:01:11.341040 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.341010    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"]
Oct 30 13:01:11.344962 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.344940    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.377418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.374042    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-87f1b5-openshift-default-dockercfg-f4wp5\""
Oct 30 13:01:11.379779 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.379756    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"]
Oct 30 13:01:11.430701 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.430674    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-token\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.430853 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.430731    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnzxt\" (UniqueName: \"kubernetes.io/projected/cee173cf-090c-42b7-8427-36e58cc4f3ba-kube-api-access-wnzxt\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.430853 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.430814    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-podinfo\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.430952 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.430940    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-credential-socket\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.430988 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.430965    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee173cf-090c-42b7-8427-36e58cc4f3ba-istiod-ca-cert\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.431040 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.431006    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-workload-certs\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.431091 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.431065    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-data\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.431221 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.431204    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-envoy\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.431322 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.431308    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-workload-socket\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.533502 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.533458    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-workload-socket\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.533875 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.533854    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-token\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.534027 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.534010    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnzxt\" (UniqueName: \"kubernetes.io/projected/cee173cf-090c-42b7-8427-36e58cc4f3ba-kube-api-access-wnzxt\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.534162 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.534147    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-podinfo\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.534354 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.534339    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-credential-socket\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.534496 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.534480    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee173cf-090c-42b7-8427-36e58cc4f3ba-istiod-ca-cert\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.538520 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.538491    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-workload-certs\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.538695 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.538677    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-data\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.538871 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.538840    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-envoy\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.572423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.533897    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-workload-socket\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.572423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.553763    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-credential-socket\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.572423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.554454    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee173cf-090c-42b7-8427-36e58cc4f3ba-istiod-ca-cert\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.572423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.554718    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-workload-certs\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.572423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.554955    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-data\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.573430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.573308    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-token\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.574249 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.573966    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-podinfo\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.574249 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.574194    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-envoy\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.599467 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.599117    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnzxt\" (UniqueName: \"kubernetes.io/projected/cee173cf-090c-42b7-8427-36e58cc4f3ba-kube-api-access-wnzxt\") pod \"e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") " pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.674415 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.669029    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:11.703476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.696329    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"
Oct 30 13:01:11.703476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.700561    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"
Oct 30 13:01:11.761597 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.761560    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2" event={"ID":"4d021c3e-5871-4c5a-aa65-0244db8501de","Type":"ContainerStarted","Data":"4dd69895117c9bdad7613beafee2e1c5a37b70de9463e564c8f4df2c5fbc5c89"}
Oct 30 13:01:11.761883 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.761856    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2" event={"ID":"4d021c3e-5871-4c5a-aa65-0244db8501de","Type":"ContainerStarted","Data":"95d17bb336675c070d104e0d714bfff92ddcc53fcaa128a5c27e41954271c5f5"}
Oct 30 13:01:11.764041 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.764018    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"
Oct 30 13:01:11.823367 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.822705    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp" event={"ID":"d17210fa-f13e-4df6-b6c6-f2f6cbc338e3","Type":"ContainerStarted","Data":"b8fc9f00f0c0ba3cb3f1c17503e55c1030385efd0f717acaaf901d3626b148c9"}
Oct 30 13:01:11.823367 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.822764    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp" event={"ID":"d17210fa-f13e-4df6-b6c6-f2f6cbc338e3","Type":"ContainerStarted","Data":"283f295628543d695d8b58c8cca41ffc9a5e5714c3708cf6960a749363a8f995"}
Oct 30 13:01:11.823367 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.823266    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"
Oct 30 13:01:11.840278 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.839174    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2" podStartSLOduration=1.839156613 podStartE2EDuration="1.839156613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:11.834789652 +0000 UTC m=+1460.359073873" watchObservedRunningTime="2026-10-30 13:01:11.839156613 +0000 UTC m=+1460.363440834"
Oct 30 13:01:11.874413 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:11.874299    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp" podStartSLOduration=1.874280505 podStartE2EDuration="1.874280505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:11.873638923 +0000 UTC m=+1460.397923144" watchObservedRunningTime="2026-10-30 13:01:11.874280505 +0000 UTC m=+1460.398564725"
Oct 30 13:01:12.124343 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:01:12.124305    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee173cf_090c_42b7_8427_36e58cc4f3ba.slice/crio-943c2fb08464fa07d2335a034a9499e275baafb602f766b0a94fad050f6bf639 WatchSource:0}: Error finding container 943c2fb08464fa07d2335a034a9499e275baafb602f766b0a94fad050f6bf639: Status 404 returned error can't find the container with id 943c2fb08464fa07d2335a034a9499e275baafb602f766b0a94fad050f6bf639
Oct 30 13:01:12.124816 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:12.124762    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"]
Oct 30 13:01:12.127304 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:12.127269    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:01:12.127418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:12.127348    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:01:12.127418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:12.127411    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:01:12.832307 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:12.832275    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92" event={"ID":"cee173cf-090c-42b7-8427-36e58cc4f3ba","Type":"ContainerStarted","Data":"cc2a500ec16ebe534e04419d11c56dd3f5eb9f695c2782752a8f4390b4ca697f"}
Oct 30 13:01:12.832821 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:12.832325    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92" event={"ID":"cee173cf-090c-42b7-8427-36e58cc4f3ba","Type":"ContainerStarted","Data":"943c2fb08464fa07d2335a034a9499e275baafb602f766b0a94fad050f6bf639"}
Oct 30 13:01:12.895350 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:12.895303    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92" podStartSLOduration=1.895290098 podStartE2EDuration="1.895290098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:12.893250121 +0000 UTC m=+1461.417534341" watchObservedRunningTime="2026-10-30 13:01:12.895290098 +0000 UTC m=+1461.419574318"
Oct 30 13:01:13.682457 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:13.682425    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:13.688843 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:13.688820    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:13.840299 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:13.840263    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:13.841028 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:13.841008    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:01:15.097179 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:01:15.097148    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:01:17.414369 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:17.408058    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"]
Oct 30 13:01:17.527304 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:17.527281    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"]
Oct 30 13:01:17.529733 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:17.529710    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"
Oct 30 13:01:17.540563 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:17.540535    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9bc72c-02a4-4edb-8534-f3b5d62a3882-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm\" (UID: \"fb9bc72c-02a4-4edb-8534-f3b5d62a3882\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"
Oct 30 13:01:17.540758 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:17.540591    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkdqr\" (UniqueName: \"kubernetes.io/projected/fb9bc72c-02a4-4edb-8534-f3b5d62a3882-kube-api-access-lkdqr\") pod \"payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm\" (UID: \"fb9bc72c-02a4-4edb-8534-f3b5d62a3882\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"
Oct 30 13:01:17.553421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:17.547265    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-87f1b5\""
Oct 30 13:01:17.556698 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:17.556653    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-87f1b5-dockercfg-2nnk8\""
Oct 30 13:01:17.642596 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:17.642548    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9bc72c-02a4-4edb-8534-f3b5d62a3882-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm\" (UID: \"fb9bc72c-02a4-4edb-8534-f3b5d62a3882\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"
Oct 30 13:01:17.642814 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:17.642651    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkdqr\" (UniqueName: \"kubernetes.io/projected/fb9bc72c-02a4-4edb-8534-f3b5d62a3882-kube-api-access-lkdqr\") pod \"payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm\" (UID: \"fb9bc72c-02a4-4edb-8534-f3b5d62a3882\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"
Oct 30 13:01:17.643282 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:17.643256    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9bc72c-02a4-4edb-8534-f3b5d62a3882-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm\" (UID: \"fb9bc72c-02a4-4edb-8534-f3b5d62a3882\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"
Oct 30 13:01:17.663240 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:17.663211    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkdqr\" (UniqueName: \"kubernetes.io/projected/fb9bc72c-02a4-4edb-8534-f3b5d62a3882-kube-api-access-lkdqr\") pod \"payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm\" (UID: \"fb9bc72c-02a4-4edb-8534-f3b5d62a3882\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"
Oct 30 13:01:17.862601 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:17.862575    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"
Oct 30 13:01:18.208193 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.208153    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"]
Oct 30 13:01:18.208588 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:01:18.208561    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9bc72c_02a4_4edb_8534_f3b5d62a3882.slice/crio-c90da873000b35b212976306a5b7332b410a0c25cea10b23e34aa28388c268b2 WatchSource:0}: Error finding container c90da873000b35b212976306a5b7332b410a0c25cea10b23e34aa28388c268b2: Status 404 returned error can't find the container with id c90da873000b35b212976306a5b7332b410a0c25cea10b23e34aa28388c268b2
Oct 30 13:01:18.475551 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.470831    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"]
Oct 30 13:01:18.590436 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.588719    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"]
Oct 30 13:01:18.590436 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.589190    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"
Oct 30 13:01:18.592580 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.591997    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b\" (UID: \"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"
Oct 30 13:01:18.592580 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.592070    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b\" (UID: \"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"
Oct 30 13:01:18.592580 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.592172    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvmdk\" (UniqueName: \"kubernetes.io/projected/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-kube-api-access-mvmdk\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b\" (UID: \"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"
Oct 30 13:01:18.604679 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.604656    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:01:18.604826 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.604694    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:01:18.634898 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.634868    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:01:18.635090 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.635016    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:01:18.694150 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.694117    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b\" (UID: \"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"
Oct 30 13:01:18.694354 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.694202    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b\" (UID: \"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"
Oct 30 13:01:18.694354 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.694261    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvmdk\" (UniqueName: \"kubernetes.io/projected/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-kube-api-access-mvmdk\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b\" (UID: \"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"
Oct 30 13:01:18.694850 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.694823    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b\" (UID: \"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"
Oct 30 13:01:18.696769 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.696741    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b\" (UID: \"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"
Oct 30 13:01:18.713644 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.713619    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvmdk\" (UniqueName: \"kubernetes.io/projected/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-kube-api-access-mvmdk\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b\" (UID: \"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"
Oct 30 13:01:18.906687 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:18.906660    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"
Oct 30 13:01:19.032806 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:19.032764    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm" event={"ID":"fb9bc72c-02a4-4edb-8534-f3b5d62a3882","Type":"ContainerStarted","Data":"aa3cf4fb19f4a187d55cf319ca801c98c854f13f83108d6decd92803efd718c2"}
Oct 30 13:01:19.032975 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:19.032832    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm" event={"ID":"fb9bc72c-02a4-4edb-8534-f3b5d62a3882","Type":"ContainerStarted","Data":"c90da873000b35b212976306a5b7332b410a0c25cea10b23e34aa28388c268b2"}
Oct 30 13:01:19.032975 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:19.032860    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"
Oct 30 13:01:19.079683 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:19.079620    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm" podStartSLOduration=2.079600152 podStartE2EDuration="2.079600152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:19.073475275 +0000 UTC m=+1467.597759491" watchObservedRunningTime="2026-10-30 13:01:19.079600152 +0000 UTC m=+1467.603884371"
Oct 30 13:01:19.256670 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:19.246067    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"]
Oct 30 13:01:20.042038 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:20.042004    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b" event={"ID":"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a","Type":"ContainerStarted","Data":"a2e617ffed22f140849847408b9672fbff91594daa255e642b304fdd6b309369"}
Oct 30 13:01:20.042518 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:20.042064    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b" event={"ID":"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a","Type":"ContainerStarted","Data":"796ca9d4d4659f3b72a8fb34306c7ecf566eace0d049da59b49696c65dc29688"}
Oct 30 13:01:20.083216 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:20.083162    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b" podStartSLOduration=2.083147958 podStartE2EDuration="2.083147958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:20.080950769 +0000 UTC m=+1468.605235001" watchObservedRunningTime="2026-10-30 13:01:20.083147958 +0000 UTC m=+1468.607432177"
Oct 30 13:01:21.050798 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:21.050767    5837 generic.go:356] "Generic (PLEG): container finished" podID="e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a" containerID="a2e617ffed22f140849847408b9672fbff91594daa255e642b304fdd6b309369" exitCode=0
Oct 30 13:01:21.051174 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:21.050850    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b" event={"ID":"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a","Type":"ContainerDied","Data":"a2e617ffed22f140849847408b9672fbff91594daa255e642b304fdd6b309369"}
Oct 30 13:01:22.250682 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.250657    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"
Oct 30 13:01:22.285710 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.285682    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-maas-api-openshift-ca\") pod \"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a\" (UID: \"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a\") "
Oct 30 13:01:22.286051 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.285821    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-maas-api-xks-ca\") pod \"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a\" (UID: \"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a\") "
Oct 30 13:01:22.286051 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.285855    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-kube-api-access-mvmdk\" (UniqueName: \"kubernetes.io/projected/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-kube-api-access-mvmdk\") pod \"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a\" (UID: \"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a\") "
Oct 30 13:01:22.286349 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.286200    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-maas-api-openshift-ca" pod "e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a" (UID: "e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:22.286690 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.286667    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:22.288585 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.288558    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-maas-api-xks-ca" pod "e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a" (UID: "e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:22.288896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.288871    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-kube-api-access-mvmdk" pod "e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a" (UID: "e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a"). InnerVolumeSpecName "kube-api-access-mvmdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:22.340781 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.340700    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"]
Oct 30 13:01:22.341512 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.341492    5837 cpu_manager.go:414] "containerMap: removing container" podUID="e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a" containerName="revoke-keys"
Oct 30 13:01:22.341619 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.341513    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a" containerName="revoke-keys"
Oct 30 13:01:22.341726 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.341713    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a" containerName="revoke-keys"
Oct 30 13:01:22.344652 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.344630    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"
Oct 30 13:01:22.368414 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.360894    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a81a48-dockercfg-95bdh\""
Oct 30 13:01:22.372920 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.372605    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"]
Oct 30 13:01:22.389254 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.389212    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtzmn\" (UniqueName: \"kubernetes.io/projected/f03a6158-3836-4f91-b4b7-98ea98c2fb90-kube-api-access-mtzmn\") pod \"payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5\" (UID: \"f03a6158-3836-4f91-b4b7-98ea98c2fb90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"
Oct 30 13:01:22.389469 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.389265    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03a6158-3836-4f91-b4b7-98ea98c2fb90-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5\" (UID: \"f03a6158-3836-4f91-b4b7-98ea98c2fb90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"
Oct 30 13:01:22.389772 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.389742    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:22.389885 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.389864    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvmdk\" (UniqueName: \"kubernetes.io/projected/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a-kube-api-access-mvmdk\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:22.409844 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.409812    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a81a48\""
Oct 30 13:01:22.548504 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.499660    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtzmn\" (UniqueName: \"kubernetes.io/projected/f03a6158-3836-4f91-b4b7-98ea98c2fb90-kube-api-access-mtzmn\") pod \"payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5\" (UID: \"f03a6158-3836-4f91-b4b7-98ea98c2fb90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"
Oct 30 13:01:22.550671 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.550620    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03a6158-3836-4f91-b4b7-98ea98c2fb90-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5\" (UID: \"f03a6158-3836-4f91-b4b7-98ea98c2fb90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"
Oct 30 13:01:22.560916 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.560880    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03a6158-3836-4f91-b4b7-98ea98c2fb90-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5\" (UID: \"f03a6158-3836-4f91-b4b7-98ea98c2fb90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"
Oct 30 13:01:22.561785 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.561758    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtzmn\" (UniqueName: \"kubernetes.io/projected/f03a6158-3836-4f91-b4b7-98ea98c2fb90-kube-api-access-mtzmn\") pod \"payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5\" (UID: \"f03a6158-3836-4f91-b4b7-98ea98c2fb90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"
Oct 30 13:01:22.680946 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.680847    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"
Oct 30 13:01:22.834523 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:22.834489    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"
Oct 30 13:01:22.996102 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:01:22.996064    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03a6158_3836_4f91_b4b7_98ea98c2fb90.slice/crio-29c9cf3698caea17d47e4b2c28f34b2cdbf67943e671308d2f26e14d0794f187 WatchSource:0}: Error finding container 29c9cf3698caea17d47e4b2c28f34b2cdbf67943e671308d2f26e14d0794f187: Status 404 returned error can't find the container with id 29c9cf3698caea17d47e4b2c28f34b2cdbf67943e671308d2f26e14d0794f187
Oct 30 13:01:23.000185 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:23.000156    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"]
Oct 30 13:01:23.068340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:23.068318    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"
Oct 30 13:01:23.068340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:23.068314    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b" event={"ID":"e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a","Type":"ContainerDied","Data":"796ca9d4d4659f3b72a8fb34306c7ecf566eace0d049da59b49696c65dc29688"}
Oct 30 13:01:23.068576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:23.068374    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="796ca9d4d4659f3b72a8fb34306c7ecf566eace0d049da59b49696c65dc29688"
Oct 30 13:01:23.069942 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:23.069920    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5" event={"ID":"f03a6158-3836-4f91-b4b7-98ea98c2fb90","Type":"ContainerStarted","Data":"29c9cf3698caea17d47e4b2c28f34b2cdbf67943e671308d2f26e14d0794f187"}
Oct 30 13:01:23.842558 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:23.842535    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"
Oct 30 13:01:24.083276 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:24.083235    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5" event={"ID":"f03a6158-3836-4f91-b4b7-98ea98c2fb90","Type":"ContainerStarted","Data":"09fc1b0a29696cafa053bb787942ad53707260e6d716c65c9894fe27d318c156"}
Oct 30 13:01:24.083631 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:24.083585    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"
Oct 30 13:01:24.125173 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:24.125059    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5" podStartSLOduration=2.125040651 podStartE2EDuration="2.125040651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:24.122584495 +0000 UTC m=+1472.646868785" watchObservedRunningTime="2026-10-30 13:01:24.125040651 +0000 UTC m=+1472.649324910"
Oct 30 13:01:28.310389 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.310356    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"]
Oct 30 13:01:28.314225 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.314207    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"
Oct 30 13:01:28.332995 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.332971    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"]
Oct 30 13:01:28.341289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.340350    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:01:28.341289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.340371    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:01:28.341289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.340362    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:01:28.341289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.340446    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:01:28.454983 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.454953    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tjkg\" (UniqueName: \"kubernetes.io/projected/fc4086b7-dead-4f8f-a5c1-86316e690a02-kube-api-access-9tjkg\") pod \"maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl\" (UID: \"fc4086b7-dead-4f8f-a5c1-86316e690a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"
Oct 30 13:01:28.455177 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.455005    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4086b7-dead-4f8f-a5c1-86316e690a02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl\" (UID: \"fc4086b7-dead-4f8f-a5c1-86316e690a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"
Oct 30 13:01:28.455284 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.455260    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4086b7-dead-4f8f-a5c1-86316e690a02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl\" (UID: \"fc4086b7-dead-4f8f-a5c1-86316e690a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"
Oct 30 13:01:28.556643 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.556613    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4086b7-dead-4f8f-a5c1-86316e690a02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl\" (UID: \"fc4086b7-dead-4f8f-a5c1-86316e690a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"
Oct 30 13:01:28.556786 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.556703    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tjkg\" (UniqueName: \"kubernetes.io/projected/fc4086b7-dead-4f8f-a5c1-86316e690a02-kube-api-access-9tjkg\") pod \"maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl\" (UID: \"fc4086b7-dead-4f8f-a5c1-86316e690a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"
Oct 30 13:01:28.556786 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.556756    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4086b7-dead-4f8f-a5c1-86316e690a02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl\" (UID: \"fc4086b7-dead-4f8f-a5c1-86316e690a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"
Oct 30 13:01:28.557231 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.557202    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4086b7-dead-4f8f-a5c1-86316e690a02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl\" (UID: \"fc4086b7-dead-4f8f-a5c1-86316e690a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"
Oct 30 13:01:28.559216 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.559195    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4086b7-dead-4f8f-a5c1-86316e690a02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl\" (UID: \"fc4086b7-dead-4f8f-a5c1-86316e690a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"
Oct 30 13:01:28.575609 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.575583    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tjkg\" (UniqueName: \"kubernetes.io/projected/fc4086b7-dead-4f8f-a5c1-86316e690a02-kube-api-access-9tjkg\") pod \"maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl\" (UID: \"fc4086b7-dead-4f8f-a5c1-86316e690a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"
Oct 30 13:01:28.627869 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.627811    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"
Oct 30 13:01:28.927831 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:28.927798    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"]
Oct 30 13:01:28.929033 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:01:28.929004    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4086b7_dead_4f8f_a5c1_86316e690a02.slice/crio-9750e91edb5f0ac8557a42cedf464a6c5358e8f7bfb8d80a29db757bf8ffac28 WatchSource:0}: Error finding container 9750e91edb5f0ac8557a42cedf464a6c5358e8f7bfb8d80a29db757bf8ffac28: Status 404 returned error can't find the container with id 9750e91edb5f0ac8557a42cedf464a6c5358e8f7bfb8d80a29db757bf8ffac28
Oct 30 13:01:29.147223 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:29.147190    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl" event={"ID":"fc4086b7-dead-4f8f-a5c1-86316e690a02","Type":"ContainerStarted","Data":"9750e91edb5f0ac8557a42cedf464a6c5358e8f7bfb8d80a29db757bf8ffac28"}
Oct 30 13:01:30.046194 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:30.046164    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"
Oct 30 13:01:30.157589 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:30.157510    5837 generic.go:356] "Generic (PLEG): container finished" podID="fc4086b7-dead-4f8f-a5c1-86316e690a02" containerID="bf518544cbbe5cb7f321b027352494e6e181a520960d6ccc7189f1c73a013b84" exitCode=0
Oct 30 13:01:30.157589 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:30.157552    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl" event={"ID":"fc4086b7-dead-4f8f-a5c1-86316e690a02","Type":"ContainerDied","Data":"bf518544cbbe5cb7f321b027352494e6e181a520960d6ccc7189f1c73a013b84"}
Oct 30 13:01:31.323801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:31.323772    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"
Oct 30 13:01:31.398143 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:31.398115    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc4086b7-dead-4f8f-a5c1-86316e690a02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4086b7-dead-4f8f-a5c1-86316e690a02-maas-api-openshift-ca\") pod \"fc4086b7-dead-4f8f-a5c1-86316e690a02\" (UID: \"fc4086b7-dead-4f8f-a5c1-86316e690a02\") "
Oct 30 13:01:31.398305 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:31.398265    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4086b7-dead-4f8f-a5c1-86316e690a02-kube-api-access-9tjkg\" (UniqueName: \"kubernetes.io/projected/fc4086b7-dead-4f8f-a5c1-86316e690a02-kube-api-access-9tjkg\") pod \"fc4086b7-dead-4f8f-a5c1-86316e690a02\" (UID: \"fc4086b7-dead-4f8f-a5c1-86316e690a02\") "
Oct 30 13:01:31.398407 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:31.398381    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc4086b7-dead-4f8f-a5c1-86316e690a02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4086b7-dead-4f8f-a5c1-86316e690a02-maas-api-xks-ca\") pod \"fc4086b7-dead-4f8f-a5c1-86316e690a02\" (UID: \"fc4086b7-dead-4f8f-a5c1-86316e690a02\") "
Oct 30 13:01:31.398948 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:31.398587    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4086b7-dead-4f8f-a5c1-86316e690a02-maas-api-openshift-ca" pod "fc4086b7-dead-4f8f-a5c1-86316e690a02" (UID: "fc4086b7-dead-4f8f-a5c1-86316e690a02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:31.399080 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:31.399060    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4086b7-dead-4f8f-a5c1-86316e690a02-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:31.400795 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:31.400765    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4086b7-dead-4f8f-a5c1-86316e690a02-kube-api-access-9tjkg" pod "fc4086b7-dead-4f8f-a5c1-86316e690a02" (UID: "fc4086b7-dead-4f8f-a5c1-86316e690a02"). InnerVolumeSpecName "kube-api-access-9tjkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:31.400951 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:31.400830    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc4086b7-dead-4f8f-a5c1-86316e690a02-maas-api-xks-ca" pod "fc4086b7-dead-4f8f-a5c1-86316e690a02" (UID: "fc4086b7-dead-4f8f-a5c1-86316e690a02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:31.506432 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:31.502490    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4086b7-dead-4f8f-a5c1-86316e690a02-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:31.506432 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:31.502530    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tjkg\" (UniqueName: \"kubernetes.io/projected/fc4086b7-dead-4f8f-a5c1-86316e690a02-kube-api-access-9tjkg\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:32.173058 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:32.173030    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"
Oct 30 13:01:32.173261 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:32.173061    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl" event={"ID":"fc4086b7-dead-4f8f-a5c1-86316e690a02","Type":"ContainerDied","Data":"9750e91edb5f0ac8557a42cedf464a6c5358e8f7bfb8d80a29db757bf8ffac28"}
Oct 30 13:01:32.173261 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:32.173109    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9750e91edb5f0ac8557a42cedf464a6c5358e8f7bfb8d80a29db757bf8ffac28"
Oct 30 13:01:35.110366 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.110333    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"
Oct 30 13:01:35.528310 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.528280    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"]
Oct 30 13:01:35.530371 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.530332    5837 cpu_manager.go:414] "containerMap: removing container" podUID="fc4086b7-dead-4f8f-a5c1-86316e690a02" containerName="revoke-keys"
Oct 30 13:01:35.530371 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.530372    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4086b7-dead-4f8f-a5c1-86316e690a02" containerName="revoke-keys"
Oct 30 13:01:35.531166 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.530903    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4086b7-dead-4f8f-a5c1-86316e690a02" containerName="revoke-keys"
Oct 30 13:01:35.536576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.536473    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"
Oct 30 13:01:35.550861 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.550670    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"]
Oct 30 13:01:35.558826 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.558798    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/431d0ac1-5825-40c3-af3b-031fa5726ffa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6\" (UID: \"431d0ac1-5825-40c3-af3b-031fa5726ffa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"
Oct 30 13:01:35.559223 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.559204    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-685tq\" (UniqueName: \"kubernetes.io/projected/431d0ac1-5825-40c3-af3b-031fa5726ffa-kube-api-access-685tq\") pod \"payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6\" (UID: \"431d0ac1-5825-40c3-af3b-031fa5726ffa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"
Oct 30 13:01:35.566114 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.566069    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e74c27\""
Oct 30 13:01:35.566688 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.566666    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e74c27-dockercfg-l9fbx\""
Oct 30 13:01:35.662014 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.661975    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/431d0ac1-5825-40c3-af3b-031fa5726ffa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6\" (UID: \"431d0ac1-5825-40c3-af3b-031fa5726ffa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"
Oct 30 13:01:35.662303 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.662139    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-685tq\" (UniqueName: \"kubernetes.io/projected/431d0ac1-5825-40c3-af3b-031fa5726ffa-kube-api-access-685tq\") pod \"payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6\" (UID: \"431d0ac1-5825-40c3-af3b-031fa5726ffa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"
Oct 30 13:01:35.663028 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.663003    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/431d0ac1-5825-40c3-af3b-031fa5726ffa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6\" (UID: \"431d0ac1-5825-40c3-af3b-031fa5726ffa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"
Oct 30 13:01:35.682202 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.682168    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-685tq\" (UniqueName: \"kubernetes.io/projected/431d0ac1-5825-40c3-af3b-031fa5726ffa-kube-api-access-685tq\") pod \"payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6\" (UID: \"431d0ac1-5825-40c3-af3b-031fa5726ffa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"
Oct 30 13:01:35.858858 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:35.858779    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"
Oct 30 13:01:36.223417 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:36.222491    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"]
Oct 30 13:01:36.223417 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:01:36.222526    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431d0ac1_5825_40c3_af3b_031fa5726ffa.slice/crio-3345911042a263fb5b98edf2af12cc064c5f76734df76697e73aa224d3c15d4e WatchSource:0}: Error finding container 3345911042a263fb5b98edf2af12cc064c5f76734df76697e73aa224d3c15d4e: Status 404 returned error can't find the container with id 3345911042a263fb5b98edf2af12cc064c5f76734df76697e73aa224d3c15d4e
Oct 30 13:01:36.260017 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:36.259963    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6" event={"ID":"431d0ac1-5825-40c3-af3b-031fa5726ffa","Type":"ContainerStarted","Data":"3345911042a263fb5b98edf2af12cc064c5f76734df76697e73aa224d3c15d4e"}
Oct 30 13:01:37.270754 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:37.270714    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6" event={"ID":"431d0ac1-5825-40c3-af3b-031fa5726ffa","Type":"ContainerStarted","Data":"07cf213b6b60177226f9dc16983a9fda3b5592d98e5f2c6f550d2e56dd6fdd2c"}
Oct 30 13:01:37.271227 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:37.270921    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"
Oct 30 13:01:37.310490 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:37.310444    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6" podStartSLOduration=2.3104285239999998 podStartE2EDuration="2.310428524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:37.307615987 +0000 UTC m=+1485.831900206" watchObservedRunningTime="2026-10-30 13:01:37.310428524 +0000 UTC m=+1485.834712741"
Oct 30 13:01:41.190309 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.190272    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"]
Oct 30 13:01:41.191021 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.190532    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj" podUID="d86d0755-fbd0-4714-9572-3612ed5198b2" containerName="payload-processing" containerID="cri-o://0d5af89b5bfd805952f47d149f2c48fba1eb1bc4fe42e1e43ea602eaf6c8c291" gracePeriod=30
Oct 30 13:01:41.223633 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.222567    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"]
Oct 30 13:01:41.223633 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.222812    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw" podUID="85c25fff-6919-42ac-9455-f15aa3988883" containerName="payload-pre-processing" containerID="cri-o://f7fe027a22db1c01db4142024e438df2f2813e426d14ad1d71e572a662045bcb" gracePeriod=30
Oct 30 13:01:41.280790 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.280756    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw" podUID="85c25fff-6919-42ac-9455-f15aa3988883" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.155:9004: connect: connection refused"
Oct 30 13:01:41.280790 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.280769    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj" podUID="d86d0755-fbd0-4714-9572-3612ed5198b2" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.156:9004: connect: connection refused"
Oct 30 13:01:41.693142 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.693092    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"
Oct 30 13:01:41.697602 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.697506    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"
Oct 30 13:01:41.805576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.800355    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c25fff-6919-42ac-9455-f15aa3988883-kube-api-access-zht7j\" (UniqueName: \"kubernetes.io/projected/85c25fff-6919-42ac-9455-f15aa3988883-kube-api-access-zht7j\") pod \"85c25fff-6919-42ac-9455-f15aa3988883\" (UID: \"85c25fff-6919-42ac-9455-f15aa3988883\") "
Oct 30 13:01:41.805576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.801109    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d86d0755-fbd0-4714-9572-3612ed5198b2-kube-api-access-m228t\" (UniqueName: \"kubernetes.io/projected/d86d0755-fbd0-4714-9572-3612ed5198b2-kube-api-access-m228t\") pod \"d86d0755-fbd0-4714-9572-3612ed5198b2\" (UID: \"d86d0755-fbd0-4714-9572-3612ed5198b2\") "
Oct 30 13:01:41.805576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.801157    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d86d0755-fbd0-4714-9572-3612ed5198b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86d0755-fbd0-4714-9572-3612ed5198b2-plugins-config-volume\") pod \"d86d0755-fbd0-4714-9572-3612ed5198b2\" (UID: \"d86d0755-fbd0-4714-9572-3612ed5198b2\") "
Oct 30 13:01:41.805576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.801191    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85c25fff-6919-42ac-9455-f15aa3988883-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c25fff-6919-42ac-9455-f15aa3988883-plugins-config-volume\") pod \"85c25fff-6919-42ac-9455-f15aa3988883\" (UID: \"85c25fff-6919-42ac-9455-f15aa3988883\") "
Oct 30 13:01:41.805576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.802259    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c25fff-6919-42ac-9455-f15aa3988883-plugins-config-volume" pod "85c25fff-6919-42ac-9455-f15aa3988883" (UID: "85c25fff-6919-42ac-9455-f15aa3988883"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:41.805576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.802665    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86d0755-fbd0-4714-9572-3612ed5198b2-plugins-config-volume" pod "d86d0755-fbd0-4714-9572-3612ed5198b2" (UID: "d86d0755-fbd0-4714-9572-3612ed5198b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:41.813807 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.813761    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c25fff-6919-42ac-9455-f15aa3988883-kube-api-access-zht7j" pod "85c25fff-6919-42ac-9455-f15aa3988883" (UID: "85c25fff-6919-42ac-9455-f15aa3988883"). InnerVolumeSpecName "kube-api-access-zht7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:41.815429 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.815384    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86d0755-fbd0-4714-9572-3612ed5198b2-kube-api-access-m228t" pod "d86d0755-fbd0-4714-9572-3612ed5198b2" (UID: "d86d0755-fbd0-4714-9572-3612ed5198b2"). InnerVolumeSpecName "kube-api-access-m228t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:41.903166 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.903126    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m228t\" (UniqueName: \"kubernetes.io/projected/d86d0755-fbd0-4714-9572-3612ed5198b2-kube-api-access-m228t\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:41.903166 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.903162    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86d0755-fbd0-4714-9572-3612ed5198b2-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:41.903166 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.903172    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c25fff-6919-42ac-9455-f15aa3988883-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:41.903424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.903180    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zht7j\" (UniqueName: \"kubernetes.io/projected/85c25fff-6919-42ac-9455-f15aa3988883-kube-api-access-zht7j\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:41.952507 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.952476    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"]
Oct 30 13:01:41.957114 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:41.957087    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1222e-xnz7b"]
Oct 30 13:01:42.313411 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.313365    5837 generic.go:356] "Generic (PLEG): container finished" podID="d86d0755-fbd0-4714-9572-3612ed5198b2" containerID="0d5af89b5bfd805952f47d149f2c48fba1eb1bc4fe42e1e43ea602eaf6c8c291" exitCode=0
Oct 30 13:01:42.313828 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.313470    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"
Oct 30 13:01:42.313828 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.313517    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj" event={"ID":"d86d0755-fbd0-4714-9572-3612ed5198b2","Type":"ContainerDied","Data":"0d5af89b5bfd805952f47d149f2c48fba1eb1bc4fe42e1e43ea602eaf6c8c291"}
Oct 30 13:01:42.313828 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.313575    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj" event={"ID":"d86d0755-fbd0-4714-9572-3612ed5198b2","Type":"ContainerDied","Data":"70c89145770aa97c306d338bb7f7d9a64debff5fd34f5beed7081d1fac33e714"}
Oct 30 13:01:42.313828 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.313607    5837 scope.go:122] "RemoveContainer" containerID="0d5af89b5bfd805952f47d149f2c48fba1eb1bc4fe42e1e43ea602eaf6c8c291"
Oct 30 13:01:42.315176 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.315095    5837 generic.go:356] "Generic (PLEG): container finished" podID="85c25fff-6919-42ac-9455-f15aa3988883" containerID="f7fe027a22db1c01db4142024e438df2f2813e426d14ad1d71e572a662045bcb" exitCode=0
Oct 30 13:01:42.315323 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.315201    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"
Oct 30 13:01:42.315323 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.315160    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw" event={"ID":"85c25fff-6919-42ac-9455-f15aa3988883","Type":"ContainerDied","Data":"f7fe027a22db1c01db4142024e438df2f2813e426d14ad1d71e572a662045bcb"}
Oct 30 13:01:42.315323 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.315296    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw" event={"ID":"85c25fff-6919-42ac-9455-f15aa3988883","Type":"ContainerDied","Data":"214f612afffe3e80f9e662f5ce30c2dd20aaff07359bcd6b3cfddd6d4d89574a"}
Oct 30 13:01:42.324275 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.324256    5837 scope.go:122] "RemoveContainer" containerID="0d5af89b5bfd805952f47d149f2c48fba1eb1bc4fe42e1e43ea602eaf6c8c291"
Oct 30 13:01:42.324653 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:01:42.324625    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5af89b5bfd805952f47d149f2c48fba1eb1bc4fe42e1e43ea602eaf6c8c291\": container with ID starting with 0d5af89b5bfd805952f47d149f2c48fba1eb1bc4fe42e1e43ea602eaf6c8c291 not found: ID does not exist" containerID="0d5af89b5bfd805952f47d149f2c48fba1eb1bc4fe42e1e43ea602eaf6c8c291"
Oct 30 13:01:42.324727 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.324664    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5af89b5bfd805952f47d149f2c48fba1eb1bc4fe42e1e43ea602eaf6c8c291"} err="failed to get container status \"0d5af89b5bfd805952f47d149f2c48fba1eb1bc4fe42e1e43ea602eaf6c8c291\": rpc error: code = NotFound desc = could not find container \"0d5af89b5bfd805952f47d149f2c48fba1eb1bc4fe42e1e43ea602eaf6c8c291\": container with ID starting with 0d5af89b5bfd805952f47d149f2c48fba1eb1bc4fe42e1e43ea602eaf6c8c291 not found: ID does not exist"
Oct 30 13:01:42.324727 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.324691    5837 scope.go:122] "RemoveContainer" containerID="f7fe027a22db1c01db4142024e438df2f2813e426d14ad1d71e572a662045bcb"
Oct 30 13:01:42.334456 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.334438    5837 scope.go:122] "RemoveContainer" containerID="f7fe027a22db1c01db4142024e438df2f2813e426d14ad1d71e572a662045bcb"
Oct 30 13:01:42.334709 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:01:42.334690    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7fe027a22db1c01db4142024e438df2f2813e426d14ad1d71e572a662045bcb\": container with ID starting with f7fe027a22db1c01db4142024e438df2f2813e426d14ad1d71e572a662045bcb not found: ID does not exist" containerID="f7fe027a22db1c01db4142024e438df2f2813e426d14ad1d71e572a662045bcb"
Oct 30 13:01:42.334775 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.334716    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7fe027a22db1c01db4142024e438df2f2813e426d14ad1d71e572a662045bcb"} err="failed to get container status \"f7fe027a22db1c01db4142024e438df2f2813e426d14ad1d71e572a662045bcb\": rpc error: code = NotFound desc = could not find container \"f7fe027a22db1c01db4142024e438df2f2813e426d14ad1d71e572a662045bcb\": container with ID starting with f7fe027a22db1c01db4142024e438df2f2813e426d14ad1d71e572a662045bcb not found: ID does not exist"
Oct 30 13:01:42.368469 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.368445    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"]
Oct 30 13:01:42.382829 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.382805    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f1222e-577f7f77dc-n9wtw"]
Oct 30 13:01:42.408845 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.408820    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"]
Oct 30 13:01:42.413719 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:42.413698    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f1222e-cc5b9d946-tf5pj"]
Oct 30 13:01:43.606528 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:43.606499    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85c25fff-6919-42ac-9455-f15aa3988883" path="/var/lib/kubelet/pods/85c25fff-6919-42ac-9455-f15aa3988883/volumes"
Oct 30 13:01:43.606969 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:43.606955    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d86d0755-fbd0-4714-9572-3612ed5198b2" path="/var/lib/kubelet/pods/d86d0755-fbd0-4714-9572-3612ed5198b2/volumes"
Oct 30 13:01:43.607384 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:43.607371    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a" path="/var/lib/kubelet/pods/e3f0eab5-0b9f-4b6a-a081-3b0b3a0f2c8a/volumes"
Oct 30 13:01:45.159436 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:01:45.157162    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:01:48.282703 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:48.282675    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"
Oct 30 13:01:48.418879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:48.418842    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"]
Oct 30 13:01:48.419121 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:48.419098    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerName="istio-proxy" containerID="cri-o://9147103a9b30857ae6dc9eee00084a44fd7616d425de4e8b5284008c3f87a216" gracePeriod=30
Oct 30 13:01:49.067829 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.067797    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"]
Oct 30 13:01:49.068628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.068610    5837 cpu_manager.go:414] "containerMap: removing container" podUID="85c25fff-6919-42ac-9455-f15aa3988883" containerName="payload-pre-processing"
Oct 30 13:01:49.068693 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.068630    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c25fff-6919-42ac-9455-f15aa3988883" containerName="payload-pre-processing"
Oct 30 13:01:49.068693 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.068674    5837 cpu_manager.go:414] "containerMap: removing container" podUID="d86d0755-fbd0-4714-9572-3612ed5198b2" containerName="payload-processing"
Oct 30 13:01:49.068693 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.068683    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d86d0755-fbd0-4714-9572-3612ed5198b2" containerName="payload-processing"
Oct 30 13:01:49.068936 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.068920    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c25fff-6919-42ac-9455-f15aa3988883" containerName="payload-pre-processing"
Oct 30 13:01:49.068997 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.068945    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d86d0755-fbd0-4714-9572-3612ed5198b2" containerName="payload-processing"
Oct 30 13:01:49.073000 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.072983    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"
Oct 30 13:01:49.095640 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.095598    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:01:49.103827 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.103803    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:01:49.104011 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.103904    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:01:49.104011 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.103958    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:01:49.107926 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.107902    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"]
Oct 30 13:01:49.155572 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.155534    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-896ql\" (UniqueName: \"kubernetes.io/projected/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-kube-api-access-896ql\") pod \"maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn\" (UID: \"0f9ad268-8f35-41d0-92e2-c12fe92f6db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"
Oct 30 13:01:49.155714 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.155591    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn\" (UID: \"0f9ad268-8f35-41d0-92e2-c12fe92f6db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"
Oct 30 13:01:49.155791 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.155767    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn\" (UID: \"0f9ad268-8f35-41d0-92e2-c12fe92f6db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"
Oct 30 13:01:49.257715 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.257686    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn\" (UID: \"0f9ad268-8f35-41d0-92e2-c12fe92f6db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"
Oct 30 13:01:49.257964 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.257944    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-896ql\" (UniqueName: \"kubernetes.io/projected/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-kube-api-access-896ql\") pod \"maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn\" (UID: \"0f9ad268-8f35-41d0-92e2-c12fe92f6db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"
Oct 30 13:01:49.258022 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.257995    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn\" (UID: \"0f9ad268-8f35-41d0-92e2-c12fe92f6db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"
Oct 30 13:01:49.258435 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.258388    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn\" (UID: \"0f9ad268-8f35-41d0-92e2-c12fe92f6db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"
Oct 30 13:01:49.260539 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.260517    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn\" (UID: \"0f9ad268-8f35-41d0-92e2-c12fe92f6db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"
Oct 30 13:01:49.276155 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.276136    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-896ql\" (UniqueName: \"kubernetes.io/projected/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-kube-api-access-896ql\") pod \"maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn\" (UID: \"0f9ad268-8f35-41d0-92e2-c12fe92f6db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"
Oct 30 13:01:49.389538 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.389453    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"
Oct 30 13:01:49.694102 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:49.694071    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"]
Oct 30 13:01:50.437817 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:50.437724    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn" event={"ID":"0f9ad268-8f35-41d0-92e2-c12fe92f6db0","Type":"ContainerStarted","Data":"30bf9b7ccb6270b0097c12375ecab7528102b12fe443a704664bd837662198f2"}
Oct 30 13:01:50.437817 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:50.437775    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn" event={"ID":"0f9ad268-8f35-41d0-92e2-c12fe92f6db0","Type":"ContainerStarted","Data":"07d5a91ba49aa9e63ba1e4ef4cb5c09fea00d8f45364721423e341e7b676970f"}
Oct 30 13:01:50.477933 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:50.477887    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn" podStartSLOduration=1.477872798 podStartE2EDuration="1.477872798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:50.474019563 +0000 UTC m=+1498.998303782" watchObservedRunningTime="2026-10-30 13:01:50.477872798 +0000 UTC m=+1499.002157045"
Oct 30 13:01:51.446701 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:51.446670    5837 generic.go:356] "Generic (PLEG): container finished" podID="0f9ad268-8f35-41d0-92e2-c12fe92f6db0" containerID="30bf9b7ccb6270b0097c12375ecab7528102b12fe443a704664bd837662198f2" exitCode=0
Oct 30 13:01:51.447104 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:51.446714    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn" event={"ID":"0f9ad268-8f35-41d0-92e2-c12fe92f6db0","Type":"ContainerDied","Data":"30bf9b7ccb6270b0097c12375ecab7528102b12fe443a704664bd837662198f2"}
Oct 30 13:01:51.902928 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:51.902902    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 13:01:51.903507 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:51.903485    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 13:01:52.364864 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:52.364829    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:01:52.365035 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:52.364887    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:01:52.636747 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:52.636687    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"
Oct 30 13:01:52.816205 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:52.816175    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-maas-api-openshift-ca\") pod \"0f9ad268-8f35-41d0-92e2-c12fe92f6db0\" (UID: \"0f9ad268-8f35-41d0-92e2-c12fe92f6db0\") "
Oct 30 13:01:52.816205 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:52.816211    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-maas-api-xks-ca\") pod \"0f9ad268-8f35-41d0-92e2-c12fe92f6db0\" (UID: \"0f9ad268-8f35-41d0-92e2-c12fe92f6db0\") "
Oct 30 13:01:52.816475 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:52.816435    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-kube-api-access-896ql\" (UniqueName: \"kubernetes.io/projected/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-kube-api-access-896ql\") pod \"0f9ad268-8f35-41d0-92e2-c12fe92f6db0\" (UID: \"0f9ad268-8f35-41d0-92e2-c12fe92f6db0\") "
Oct 30 13:01:52.816551 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:52.816500    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-maas-api-openshift-ca" pod "0f9ad268-8f35-41d0-92e2-c12fe92f6db0" (UID: "0f9ad268-8f35-41d0-92e2-c12fe92f6db0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:52.816976 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:52.816955    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:52.818494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:52.818468    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-maas-api-xks-ca" pod "0f9ad268-8f35-41d0-92e2-c12fe92f6db0" (UID: "0f9ad268-8f35-41d0-92e2-c12fe92f6db0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:52.818622 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:52.818593    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-kube-api-access-896ql" pod "0f9ad268-8f35-41d0-92e2-c12fe92f6db0" (UID: "0f9ad268-8f35-41d0-92e2-c12fe92f6db0"). InnerVolumeSpecName "kube-api-access-896ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:52.918327 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:52.918259    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-896ql\" (UniqueName: \"kubernetes.io/projected/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-kube-api-access-896ql\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:52.918327 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:52.918285    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9ad268-8f35-41d0-92e2-c12fe92f6db0-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:53.207319 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:53.207240    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"]
Oct 30 13:01:53.207494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:53.207466    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2" podUID="c33b2398-1559-4ffd-91a9-3d60d2a07778" containerName="payload-processing" containerID="cri-o://c3bbaddcd5283d659060964af43b085609ec98d61ede1841d0bb8e6f16be9b59" gracePeriod=30
Oct 30 13:01:53.227791 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:53.227763    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"]
Oct 30 13:01:53.228183 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:53.228147    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b" podUID="8baef900-5f0d-482f-8806-db129254c21b" containerName="payload-pre-processing" containerID="cri-o://d1c487b8fe2fa69b1686e84af33a0047327b3c036fec9e33daf9f78b41590530" gracePeriod=30
Oct 30 13:01:53.469605 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:53.469541    5837 generic.go:356] "Generic (PLEG): container finished" podID="c33b2398-1559-4ffd-91a9-3d60d2a07778" containerID="c3bbaddcd5283d659060964af43b085609ec98d61ede1841d0bb8e6f16be9b59" exitCode=0
Oct 30 13:01:53.469753 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:53.469689    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2" event={"ID":"c33b2398-1559-4ffd-91a9-3d60d2a07778","Type":"ContainerDied","Data":"c3bbaddcd5283d659060964af43b085609ec98d61ede1841d0bb8e6f16be9b59"}
Oct 30 13:01:53.471687 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:53.471661    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn" event={"ID":"0f9ad268-8f35-41d0-92e2-c12fe92f6db0","Type":"ContainerDied","Data":"07d5a91ba49aa9e63ba1e4ef4cb5c09fea00d8f45364721423e341e7b676970f"}
Oct 30 13:01:53.471764 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:53.471716    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07d5a91ba49aa9e63ba1e4ef4cb5c09fea00d8f45364721423e341e7b676970f"
Oct 30 13:01:53.471803 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:53.471794    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"
Oct 30 13:01:53.475008 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:53.474984    5837 generic.go:356] "Generic (PLEG): container finished" podID="8baef900-5f0d-482f-8806-db129254c21b" containerID="d1c487b8fe2fa69b1686e84af33a0047327b3c036fec9e33daf9f78b41590530" exitCode=0
Oct 30 13:01:53.475136 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:53.475070    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b" event={"ID":"8baef900-5f0d-482f-8806-db129254c21b","Type":"ContainerDied","Data":"d1c487b8fe2fa69b1686e84af33a0047327b3c036fec9e33daf9f78b41590530"}
Oct 30 13:01:53.828629 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:53.828589    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"
Oct 30 13:01:53.834252 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:53.834224    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"
Oct 30 13:01:53.927496 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:53.927461    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"]
Oct 30 13:01:53.971420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:53.945089    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dea6ee-rg4kl"]
Oct 30 13:01:54.006107 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.006028    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c33b2398-1559-4ffd-91a9-3d60d2a07778-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33b2398-1559-4ffd-91a9-3d60d2a07778-plugins-config-volume\") pod \"c33b2398-1559-4ffd-91a9-3d60d2a07778\" (UID: \"c33b2398-1559-4ffd-91a9-3d60d2a07778\") "
Oct 30 13:01:54.006107 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.006098    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33b2398-1559-4ffd-91a9-3d60d2a07778-kube-api-access-2qh4m\" (UniqueName: \"kubernetes.io/projected/c33b2398-1559-4ffd-91a9-3d60d2a07778-kube-api-access-2qh4m\") pod \"c33b2398-1559-4ffd-91a9-3d60d2a07778\" (UID: \"c33b2398-1559-4ffd-91a9-3d60d2a07778\") "
Oct 30 13:01:54.009092 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.008839    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33b2398-1559-4ffd-91a9-3d60d2a07778-plugins-config-volume" pod "c33b2398-1559-4ffd-91a9-3d60d2a07778" (UID: "c33b2398-1559-4ffd-91a9-3d60d2a07778"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:54.009092 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.008962    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8baef900-5f0d-482f-8806-db129254c21b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8baef900-5f0d-482f-8806-db129254c21b-plugins-config-volume\") pod \"8baef900-5f0d-482f-8806-db129254c21b\" (UID: \"8baef900-5f0d-482f-8806-db129254c21b\") "
Oct 30 13:01:54.009301 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.009165    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8baef900-5f0d-482f-8806-db129254c21b-kube-api-access-8djv7\" (UniqueName: \"kubernetes.io/projected/8baef900-5f0d-482f-8806-db129254c21b-kube-api-access-8djv7\") pod \"8baef900-5f0d-482f-8806-db129254c21b\" (UID: \"8baef900-5f0d-482f-8806-db129254c21b\") "
Oct 30 13:01:54.009813 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.009787    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8baef900-5f0d-482f-8806-db129254c21b-plugins-config-volume" pod "8baef900-5f0d-482f-8806-db129254c21b" (UID: "8baef900-5f0d-482f-8806-db129254c21b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:54.014467 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.011888    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33b2398-1559-4ffd-91a9-3d60d2a07778-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:54.014467 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.011917    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8baef900-5f0d-482f-8806-db129254c21b-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:54.020446 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.020419    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8baef900-5f0d-482f-8806-db129254c21b-kube-api-access-8djv7" pod "8baef900-5f0d-482f-8806-db129254c21b" (UID: "8baef900-5f0d-482f-8806-db129254c21b"). InnerVolumeSpecName "kube-api-access-8djv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:54.027561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.026225    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 13:01:54.028314 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.028291    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33b2398-1559-4ffd-91a9-3d60d2a07778-kube-api-access-2qh4m" pod "c33b2398-1559-4ffd-91a9-3d60d2a07778" (UID: "c33b2398-1559-4ffd-91a9-3d60d2a07778"). InnerVolumeSpecName "kube-api-access-2qh4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:54.119053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.115439    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-workload-socket\") pod \"d3aed56e-a69c-46a1-97e6-ea012d99e593\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") "
Oct 30 13:01:54.119053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.115513    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-envoy\") pod \"d3aed56e-a69c-46a1-97e6-ea012d99e593\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") "
Oct 30 13:01:54.119053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.115588    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-data\") pod \"d3aed56e-a69c-46a1-97e6-ea012d99e593\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") "
Oct 30 13:01:54.119053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.115787    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-credential-socket\") pod \"d3aed56e-a69c-46a1-97e6-ea012d99e593\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") "
Oct 30 13:01:54.119053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.115853    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3aed56e-a69c-46a1-97e6-ea012d99e593-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3aed56e-a69c-46a1-97e6-ea012d99e593-istiod-ca-cert\") pod \"d3aed56e-a69c-46a1-97e6-ea012d99e593\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") "
Oct 30 13:01:54.119053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.115919    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3aed56e-a69c-46a1-97e6-ea012d99e593-kube-api-access-9mnbs\" (UniqueName: \"kubernetes.io/projected/d3aed56e-a69c-46a1-97e6-ea012d99e593-kube-api-access-9mnbs\") pod \"d3aed56e-a69c-46a1-97e6-ea012d99e593\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") "
Oct 30 13:01:54.119053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.116006    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-podinfo\") pod \"d3aed56e-a69c-46a1-97e6-ea012d99e593\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") "
Oct 30 13:01:54.119053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.116085    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-workload-certs\") pod \"d3aed56e-a69c-46a1-97e6-ea012d99e593\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") "
Oct 30 13:01:54.119053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.116568    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-token\" (UniqueName: \"kubernetes.io/projected/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-token\") pod \"d3aed56e-a69c-46a1-97e6-ea012d99e593\" (UID: \"d3aed56e-a69c-46a1-97e6-ea012d99e593\") "
Oct 30 13:01:54.131423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.121584    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8djv7\" (UniqueName: \"kubernetes.io/projected/8baef900-5f0d-482f-8806-db129254c21b-kube-api-access-8djv7\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:54.131423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.121618    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qh4m\" (UniqueName: \"kubernetes.io/projected/c33b2398-1559-4ffd-91a9-3d60d2a07778-kube-api-access-2qh4m\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:54.131679 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.131609    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-workload-socket" pod "d3aed56e-a69c-46a1-97e6-ea012d99e593" (UID: "d3aed56e-a69c-46a1-97e6-ea012d99e593"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:01:54.150060 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.137767    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-workload-certs" pod "d3aed56e-a69c-46a1-97e6-ea012d99e593" (UID: "d3aed56e-a69c-46a1-97e6-ea012d99e593"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:01:54.150060 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.138235    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-credential-socket" pod "d3aed56e-a69c-46a1-97e6-ea012d99e593" (UID: "d3aed56e-a69c-46a1-97e6-ea012d99e593"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:01:54.150060 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.142493    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-data" pod "d3aed56e-a69c-46a1-97e6-ea012d99e593" (UID: "d3aed56e-a69c-46a1-97e6-ea012d99e593"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:01:54.150060 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.149576    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3aed56e-a69c-46a1-97e6-ea012d99e593-kube-api-access-9mnbs" pod "d3aed56e-a69c-46a1-97e6-ea012d99e593" (UID: "d3aed56e-a69c-46a1-97e6-ea012d99e593"). InnerVolumeSpecName "kube-api-access-9mnbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:54.150060 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.149936    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3aed56e-a69c-46a1-97e6-ea012d99e593-istiod-ca-cert" pod "d3aed56e-a69c-46a1-97e6-ea012d99e593" (UID: "d3aed56e-a69c-46a1-97e6-ea012d99e593"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:54.150657 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.150078    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-podinfo" pod "d3aed56e-a69c-46a1-97e6-ea012d99e593" (UID: "d3aed56e-a69c-46a1-97e6-ea012d99e593"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:01:54.151083 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.151051    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-token" pod "d3aed56e-a69c-46a1-97e6-ea012d99e593" (UID: "d3aed56e-a69c-46a1-97e6-ea012d99e593"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:54.151522 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.151470    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-envoy" pod "d3aed56e-a69c-46a1-97e6-ea012d99e593" (UID: "d3aed56e-a69c-46a1-97e6-ea012d99e593"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:01:54.223634 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.223600    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:54.223634 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.223624    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:54.223634 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.223638    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:54.223879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.223652    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:54.223879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.223666    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:54.223879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.223678    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3aed56e-a69c-46a1-97e6-ea012d99e593-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:54.223879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.223686    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mnbs\" (UniqueName: \"kubernetes.io/projected/d3aed56e-a69c-46a1-97e6-ea012d99e593-kube-api-access-9mnbs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:54.223879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.223694    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3aed56e-a69c-46a1-97e6-ea012d99e593-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:54.223879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.223702    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3aed56e-a69c-46a1-97e6-ea012d99e593-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:01:54.485331 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.485296    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"
Oct 30 13:01:54.485331 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.485306    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2" event={"ID":"c33b2398-1559-4ffd-91a9-3d60d2a07778","Type":"ContainerDied","Data":"f605751fa1132f15db4678a3a5a3f4506513be41378eb898971f50177d91be32"}
Oct 30 13:01:54.485608 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.485410    5837 scope.go:122] "RemoveContainer" containerID="c3bbaddcd5283d659060964af43b085609ec98d61ede1841d0bb8e6f16be9b59"
Oct 30 13:01:54.487495 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.487385    5837 generic.go:356] "Generic (PLEG): container finished" podID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerID="9147103a9b30857ae6dc9eee00084a44fd7616d425de4e8b5284008c3f87a216" exitCode=0
Oct 30 13:01:54.487635 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.487488    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" event={"ID":"d3aed56e-a69c-46a1-97e6-ea012d99e593","Type":"ContainerDied","Data":"9147103a9b30857ae6dc9eee00084a44fd7616d425de4e8b5284008c3f87a216"}
Oct 30 13:01:54.487635 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.487548    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb" event={"ID":"d3aed56e-a69c-46a1-97e6-ea012d99e593","Type":"ContainerDied","Data":"67c7e8fcde34a34f9e19afd22f3457e9b93b95ef072eac9942d5f6a4c8c9b654"}
Oct 30 13:01:54.487635 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.487478    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"
Oct 30 13:01:54.489508 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.489484    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b" event={"ID":"8baef900-5f0d-482f-8806-db129254c21b","Type":"ContainerDied","Data":"dac6988f71dac73e6e3d5e27dfb5a9eb7803ca77400b27afac0191b8f670132a"}
Oct 30 13:01:54.489508 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.489502    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"
Oct 30 13:01:54.497631 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.497614    5837 scope.go:122] "RemoveContainer" containerID="9147103a9b30857ae6dc9eee00084a44fd7616d425de4e8b5284008c3f87a216"
Oct 30 13:01:54.508529 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.508511    5837 scope.go:122] "RemoveContainer" containerID="c8da97fb89e6d59c66ff8b4144070c3d0054c9a22fdf1c723fdf9d8c888ad50a"
Oct 30 13:01:54.517994 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.517979    5837 scope.go:122] "RemoveContainer" containerID="9147103a9b30857ae6dc9eee00084a44fd7616d425de4e8b5284008c3f87a216"
Oct 30 13:01:54.518243 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:01:54.518225    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9147103a9b30857ae6dc9eee00084a44fd7616d425de4e8b5284008c3f87a216\": container with ID starting with 9147103a9b30857ae6dc9eee00084a44fd7616d425de4e8b5284008c3f87a216 not found: ID does not exist" containerID="9147103a9b30857ae6dc9eee00084a44fd7616d425de4e8b5284008c3f87a216"
Oct 30 13:01:54.518307 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.518249    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9147103a9b30857ae6dc9eee00084a44fd7616d425de4e8b5284008c3f87a216"} err="failed to get container status \"9147103a9b30857ae6dc9eee00084a44fd7616d425de4e8b5284008c3f87a216\": rpc error: code = NotFound desc = could not find container \"9147103a9b30857ae6dc9eee00084a44fd7616d425de4e8b5284008c3f87a216\": container with ID starting with 9147103a9b30857ae6dc9eee00084a44fd7616d425de4e8b5284008c3f87a216 not found: ID does not exist"
Oct 30 13:01:54.518307 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.518265    5837 scope.go:122] "RemoveContainer" containerID="c8da97fb89e6d59c66ff8b4144070c3d0054c9a22fdf1c723fdf9d8c888ad50a"
Oct 30 13:01:54.518561 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:01:54.518546    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8da97fb89e6d59c66ff8b4144070c3d0054c9a22fdf1c723fdf9d8c888ad50a\": container with ID starting with c8da97fb89e6d59c66ff8b4144070c3d0054c9a22fdf1c723fdf9d8c888ad50a not found: ID does not exist" containerID="c8da97fb89e6d59c66ff8b4144070c3d0054c9a22fdf1c723fdf9d8c888ad50a"
Oct 30 13:01:54.518614 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.518565    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8da97fb89e6d59c66ff8b4144070c3d0054c9a22fdf1c723fdf9d8c888ad50a"} err="failed to get container status \"c8da97fb89e6d59c66ff8b4144070c3d0054c9a22fdf1c723fdf9d8c888ad50a\": rpc error: code = NotFound desc = could not find container \"c8da97fb89e6d59c66ff8b4144070c3d0054c9a22fdf1c723fdf9d8c888ad50a\": container with ID starting with c8da97fb89e6d59c66ff8b4144070c3d0054c9a22fdf1c723fdf9d8c888ad50a not found: ID does not exist"
Oct 30 13:01:54.518614 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.518577    5837 scope.go:122] "RemoveContainer" containerID="d1c487b8fe2fa69b1686e84af33a0047327b3c036fec9e33daf9f78b41590530"
Oct 30 13:01:54.564033 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.564004    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"]
Oct 30 13:01:54.567874 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.567855    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dea6ee-68c5744779-vth9b"]
Oct 30 13:01:54.600328 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.600291    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"]
Oct 30 13:01:54.604229 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.604208    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f1222e-openshift-default-864954bd4c-6wlcb"]
Oct 30 13:01:54.634078 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.634052    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"]
Oct 30 13:01:54.639491 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:54.639470    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dea6ee-6cffdbb5bb-jmmb2"]
Oct 30 13:01:55.606026 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:55.605993    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8baef900-5f0d-482f-8806-db129254c21b" path="/var/lib/kubelet/pods/8baef900-5f0d-482f-8806-db129254c21b/volumes"
Oct 30 13:01:55.606449 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:55.606434    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33b2398-1559-4ffd-91a9-3d60d2a07778" path="/var/lib/kubelet/pods/c33b2398-1559-4ffd-91a9-3d60d2a07778/volumes"
Oct 30 13:01:55.606907 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:55.606893    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" path="/var/lib/kubelet/pods/d3aed56e-a69c-46a1-97e6-ea012d99e593/volumes"
Oct 30 13:01:55.607638 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:55.607623    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4086b7-dead-4f8f-a5c1-86316e690a02" path="/var/lib/kubelet/pods/fc4086b7-dead-4f8f-a5c1-86316e690a02/volumes"
Oct 30 13:01:56.828086 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:56.828050    5837 scope.go:122] "RemoveContainer" containerID="0c522777a614bccb3873f46b7c2334527f76bbf19b4c3ab69572050d305941b5"
Oct 30 13:01:56.838470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:56.838444    5837 scope.go:122] "RemoveContainer" containerID="88e4ce5581d3f6d0c663ecf2c054b1def54ca8210d292cef4a44ce9278842cef"
Oct 30 13:01:56.848008 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:56.847984    5837 scope.go:122] "RemoveContainer" containerID="897298e09ac3e463dcac2a8a7e3804fbef37647db7dba74f37e39856d29fc5ac"
Oct 30 13:01:56.857520 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:01:56.857499    5837 scope.go:122] "RemoveContainer" containerID="c1c87ab10ae795db8e04c0d58fe696b25a9799838f7635bd742bb93b0cd483cd"
Oct 30 13:02:01.482426 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:01.481589    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"]
Oct 30 13:02:01.482426 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:01.481822    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx" podUID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" containerName="istio-proxy" containerID="cri-o://f4efac654666cfd62636cf6d5b89ca8778c5f4a6675463bd5df070579054bd17" gracePeriod=30
Oct 30 13:02:01.504463 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:01.500504    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:01.913423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:01.881590    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:01.983424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:01.982479    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:02.030490 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:01.921548    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:02.030490 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:02.002767    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" podUID="48deaa90-e66d-454f-af97-1a5462424e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:02.109079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:02.109039    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:02.821874 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:02.821835    5837 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:02.845422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:02.835181    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:02.883914 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:02.850012    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:03.031529 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:02:03.031493    5837 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.118s"
Oct 30 13:02:03.105469 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.089950    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 13:02:03.105769 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.105726    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"]
Oct 30 13:02:03.131668 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.131643    5837 cpu_manager.go:414] "containerMap: removing container" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerName="istio-proxy"
Oct 30 13:02:03.131896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.131883    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerName="istio-proxy"
Oct 30 13:02:03.131992 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.131982    5837 cpu_manager.go:414] "containerMap: removing container" podUID="8baef900-5f0d-482f-8806-db129254c21b" containerName="payload-pre-processing"
Oct 30 13:02:03.132126 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.132113    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8baef900-5f0d-482f-8806-db129254c21b" containerName="payload-pre-processing"
Oct 30 13:02:03.132245 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.132235    5837 cpu_manager.go:414] "containerMap: removing container" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerName="istio-proxy"
Oct 30 13:02:03.132340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.132330    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerName="istio-proxy"
Oct 30 13:02:03.132478 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.132468    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0f9ad268-8f35-41d0-92e2-c12fe92f6db0" containerName="revoke-keys"
Oct 30 13:02:03.132570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.132561    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9ad268-8f35-41d0-92e2-c12fe92f6db0" containerName="revoke-keys"
Oct 30 13:02:03.132688 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.132679    5837 cpu_manager.go:414] "containerMap: removing container" podUID="c33b2398-1559-4ffd-91a9-3d60d2a07778" containerName="payload-processing"
Oct 30 13:02:03.132785 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.132773    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33b2398-1559-4ffd-91a9-3d60d2a07778" containerName="payload-processing"
Oct 30 13:02:03.133056 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.133046    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerName="istio-proxy"
Oct 30 13:02:03.133149 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.133140    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="8baef900-5f0d-482f-8806-db129254c21b" containerName="payload-pre-processing"
Oct 30 13:02:03.133234 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.133226    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3aed56e-a69c-46a1-97e6-ea012d99e593" containerName="istio-proxy"
Oct 30 13:02:03.133332 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.133324    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9ad268-8f35-41d0-92e2-c12fe92f6db0" containerName="revoke-keys"
Oct 30 13:02:03.133444 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.133434    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33b2398-1559-4ffd-91a9-3d60d2a07778" containerName="payload-processing"
Oct 30 13:02:03.137722 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.137674    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:03.545825 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.545791    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"]
Oct 30 13:02:03.546144 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.546125    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 13:02:03.555695 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.555664    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"
Oct 30 13:02:03.586636 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.574439    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:02:03.586858 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.582097    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:02:03.586953 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.582148    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:02:03.678469 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.677763    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:02:03.682544 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.681525    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff38ff73-fe6f-4764-89f0-c1744ad4546a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x\" (UID: \"ff38ff73-fe6f-4764-89f0-c1744ad4546a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"
Oct 30 13:02:03.682544 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.681736    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff38ff73-fe6f-4764-89f0-c1744ad4546a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x\" (UID: \"ff38ff73-fe6f-4764-89f0-c1744ad4546a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"
Oct 30 13:02:03.682544 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.681788    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrfcl\" (UniqueName: \"kubernetes.io/projected/ff38ff73-fe6f-4764-89f0-c1744ad4546a-kube-api-access-rrfcl\") pod \"maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x\" (UID: \"ff38ff73-fe6f-4764-89f0-c1744ad4546a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"
Oct 30 13:02:03.917265 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.917170    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff38ff73-fe6f-4764-89f0-c1744ad4546a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x\" (UID: \"ff38ff73-fe6f-4764-89f0-c1744ad4546a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"
Oct 30 13:02:03.918223 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.918178    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrfcl\" (UniqueName: \"kubernetes.io/projected/ff38ff73-fe6f-4764-89f0-c1744ad4546a-kube-api-access-rrfcl\") pod \"maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x\" (UID: \"ff38ff73-fe6f-4764-89f0-c1744ad4546a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"
Oct 30 13:02:03.918784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.918754    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff38ff73-fe6f-4764-89f0-c1744ad4546a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x\" (UID: \"ff38ff73-fe6f-4764-89f0-c1744ad4546a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"
Oct 30 13:02:03.920225 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:03.920191    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff38ff73-fe6f-4764-89f0-c1744ad4546a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x\" (UID: \"ff38ff73-fe6f-4764-89f0-c1744ad4546a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"
Oct 30 13:02:04.089689 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:04.089645    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:04.090069 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:04.090040    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:04.249841 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:04.154495    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:04.250931 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:04.166221    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrfcl\" (UniqueName: \"kubernetes.io/projected/ff38ff73-fe6f-4764-89f0-c1744ad4546a-kube-api-access-rrfcl\") pod \"maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x\" (UID: \"ff38ff73-fe6f-4764-89f0-c1744ad4546a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"
Oct 30 13:02:04.303827 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:04.238912    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff38ff73-fe6f-4764-89f0-c1744ad4546a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x\" (UID: \"ff38ff73-fe6f-4764-89f0-c1744ad4546a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"
Oct 30 13:02:04.396710 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:04.396670    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"
Oct 30 13:02:04.704420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:04.691537    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": context deadline exceeded"
Oct 30 13:02:04.819017 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:04.818981    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:04.819295 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:04.819263    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" podUID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:05.101422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:05.101089    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:05.264208 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:05.264167    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:05.271526 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:05.271474    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:05.276776 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:05.276750    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:02:05.514419 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:05.489542    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:05.514419 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:05.489608    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp" podUID="88a518b4-df4c-4b56-b56a-a979acf3ca15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:05.801791 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:05.801700    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:05.802053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:05.802028    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w" podUID="bb740ce8-a013-43fd-b48c-96561ca1d3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:06.070478 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:06.069732    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:06.070478 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:06.069795    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:06.101523 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:06.101483    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 13:02:06.266543 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:06.266497    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:06.277530 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:06.277482    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:02:06.691427 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:06.689870    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:06.691427 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:06.689985    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:07.118678 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:07.118631    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:07.235421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:07.231563    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"]
Oct 30 13:02:07.276420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:07.271349    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:07.301443 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:07.277924    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:07.725420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:07.709292    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:07.725420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:07.709377    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:08.120061 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:08.120008    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"]
Oct 30 13:02:08.133556 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:08.125303    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:08.338139 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:08.338083    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:02:09.026455 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.018558    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"
Oct 30 13:02:09.056703 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.056664    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"]
Oct 30 13:02:09.274605 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.274575    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf48c8b9-l7fz5\" (UID: \"a2f13e3f-c6ea-48b1-b60f-84418d3f0447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"
Oct 30 13:02:09.275221 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.275193    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf48c8b9-l7fz5\" (UID: \"a2f13e3f-c6ea-48b1-b60f-84418d3f0447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"
Oct 30 13:02:09.275479 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.275461    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69qkv\" (UniqueName: \"kubernetes.io/projected/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-kube-api-access-69qkv\") pod \"maas-api-revoke-sub-bf48c8b9-l7fz5\" (UID: \"a2f13e3f-c6ea-48b1-b60f-84418d3f0447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"
Oct 30 13:02:09.275897 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.275868    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:09.282576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:08.327373    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:02:09.543957 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.379580    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf48c8b9-l7fz5\" (UID: \"a2f13e3f-c6ea-48b1-b60f-84418d3f0447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"
Oct 30 13:02:09.547622 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.547580    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf48c8b9-l7fz5\" (UID: \"a2f13e3f-c6ea-48b1-b60f-84418d3f0447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"
Oct 30 13:02:09.547938 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.547916    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69qkv\" (UniqueName: \"kubernetes.io/projected/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-kube-api-access-69qkv\") pod \"maas-api-revoke-sub-bf48c8b9-l7fz5\" (UID: \"a2f13e3f-c6ea-48b1-b60f-84418d3f0447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"
Oct 30 13:02:09.571426 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.422350    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x" event={"ID":"ff38ff73-fe6f-4764-89f0-c1744ad4546a","Type":"ContainerStarted","Data":"568e015f0f4334b0f308303fa841907673becd1699d78d52fc5b7bfd352de807"}
Oct 30 13:02:09.571727 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.571696    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx" event={"ID":"1eb8b513-9fce-4ddd-9555-9a61cbe430dc","Type":"ContainerDied","Data":"f4efac654666cfd62636cf6d5b89ca8778c5f4a6675463bd5df070579054bd17"}
Oct 30 13:02:09.572888 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.572865    5837 scope.go:122] "RemoveContainer" containerID="637c02b83950688e877e18e8245863ba25f1074f2b2c0589ec1359c9a33a4dc3"
Oct 30 13:02:09.573223 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.510014    5837 generic.go:356] "Generic (PLEG): container finished" podID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" containerID="f4efac654666cfd62636cf6d5b89ca8778c5f4a6675463bd5df070579054bd17" exitCode=0
Oct 30 13:02:09.573472 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.564500    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:09.573630 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.533470    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:09.585420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.579305    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69qkv\" (UniqueName: \"kubernetes.io/projected/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-kube-api-access-69qkv\") pod \"maas-api-revoke-sub-bf48c8b9-l7fz5\" (UID: \"a2f13e3f-c6ea-48b1-b60f-84418d3f0447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"
Oct 30 13:02:09.612249 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.553100    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf48c8b9-l7fz5\" (UID: \"a2f13e3f-c6ea-48b1-b60f-84418d3f0447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"
Oct 30 13:02:09.612249 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.567729    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 13:02:09.612249 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.609830    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf48c8b9-l7fz5\" (UID: \"a2f13e3f-c6ea-48b1-b60f-84418d3f0447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"
Oct 30 13:02:09.687539 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.687513    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 13:02:09.699218 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:02:09.699178    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff38ff73_fe6f_4764_89f0_c1744ad4546a.slice/crio-conmon-61154274d24043eccac022f5e2bd5401e2847e6a1c5b72cdb845e740d39479ba.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:02:09.749138 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:02:09.742172    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff38ff73_fe6f_4764_89f0_c1744ad4546a.slice/crio-conmon-61154274d24043eccac022f5e2bd5401e2847e6a1c5b72cdb845e740d39479ba.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff38ff73_fe6f_4764_89f0_c1744ad4546a.slice/crio-61154274d24043eccac022f5e2bd5401e2847e6a1c5b72cdb845e740d39479ba.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:02:09.758303 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.758262    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-envoy\") pod \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") "
Oct 30 13:02:09.760064 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.760040    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-credential-socket\") pod \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") "
Oct 30 13:02:09.760273 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.760255    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istiod-ca-cert\") pod \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") "
Oct 30 13:02:09.760473 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.760441    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-workload-certs\") pod \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") "
Oct 30 13:02:09.760729 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.760700    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-kube-api-access-zslcq\" (UniqueName: \"kubernetes.io/projected/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-kube-api-access-zslcq\") pod \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") "
Oct 30 13:02:09.761344 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.761313    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-data\") pod \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") "
Oct 30 13:02:09.761576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.761539    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-podinfo\") pod \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") "
Oct 30 13:02:09.761799 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.761782    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-workload-socket\") pod \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") "
Oct 30 13:02:09.761997 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.761951    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-token\" (UniqueName: \"kubernetes.io/projected/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-token\") pod \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\" (UID: \"1eb8b513-9fce-4ddd-9555-9a61cbe430dc\") "
Oct 30 13:02:09.763655 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.763623    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-data" pod "1eb8b513-9fce-4ddd-9555-9a61cbe430dc" (UID: "1eb8b513-9fce-4ddd-9555-9a61cbe430dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:09.765297 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.764563    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-credential-socket" pod "1eb8b513-9fce-4ddd-9555-9a61cbe430dc" (UID: "1eb8b513-9fce-4ddd-9555-9a61cbe430dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:09.765297 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.764843    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istiod-ca-cert" pod "1eb8b513-9fce-4ddd-9555-9a61cbe430dc" (UID: "1eb8b513-9fce-4ddd-9555-9a61cbe430dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:09.765297 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.765065    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-workload-certs" pod "1eb8b513-9fce-4ddd-9555-9a61cbe430dc" (UID: "1eb8b513-9fce-4ddd-9555-9a61cbe430dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:09.769151 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.768717    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-workload-socket" pod "1eb8b513-9fce-4ddd-9555-9a61cbe430dc" (UID: "1eb8b513-9fce-4ddd-9555-9a61cbe430dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:09.777889 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.773478    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-kube-api-access-zslcq" pod "1eb8b513-9fce-4ddd-9555-9a61cbe430dc" (UID: "1eb8b513-9fce-4ddd-9555-9a61cbe430dc"). InnerVolumeSpecName "kube-api-access-zslcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:09.780995 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.780973    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-podinfo" pod "1eb8b513-9fce-4ddd-9555-9a61cbe430dc" (UID: "1eb8b513-9fce-4ddd-9555-9a61cbe430dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:02:09.834274 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.833869    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-envoy" pod "1eb8b513-9fce-4ddd-9555-9a61cbe430dc" (UID: "1eb8b513-9fce-4ddd-9555-9a61cbe430dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:09.864316 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.864259    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-token" pod "1eb8b513-9fce-4ddd-9555-9a61cbe430dc" (UID: "1eb8b513-9fce-4ddd-9555-9a61cbe430dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:09.884043 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.883789    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"
Oct 30 13:02:09.905784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.903626    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.905784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.903659    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.905784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.903673    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zslcq\" (UniqueName: \"kubernetes.io/projected/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-kube-api-access-zslcq\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.905784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.903687    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.905784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.903701    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.905784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.903714    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.905784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.903728    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.905784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.903740    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.905784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.903756    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb8b513-9fce-4ddd-9555-9a61cbe430dc-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.921701 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.921670    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:02:09.956879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:09.956847    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 13:02:10.344987 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:10.344952    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"]
Oct 30 13:02:10.583084 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:10.583043    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5" event={"ID":"a2f13e3f-c6ea-48b1-b60f-84418d3f0447","Type":"ContainerStarted","Data":"c572e7956ed64eb8976cdb2d16ddc04b452b562be354a78d7bef3e50026f01f8"}
Oct 30 13:02:10.612437 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:10.612104    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"
Oct 30 13:02:10.612437 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:10.612377    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx" event={"ID":"1eb8b513-9fce-4ddd-9555-9a61cbe430dc","Type":"ContainerDied","Data":"e229cf43d9a5567863ce97b13ac5bd4943e804973fcfbb5176a275c093629858"}
Oct 30 13:02:10.612708 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:10.612487    5837 scope.go:122] "RemoveContainer" containerID="f4efac654666cfd62636cf6d5b89ca8778c5f4a6675463bd5df070579054bd17"
Oct 30 13:02:10.615640 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:10.615603    5837 generic.go:356] "Generic (PLEG): container finished" podID="ff38ff73-fe6f-4764-89f0-c1744ad4546a" containerID="61154274d24043eccac022f5e2bd5401e2847e6a1c5b72cdb845e740d39479ba" exitCode=0
Oct 30 13:02:10.615759 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:10.615666    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x" event={"ID":"ff38ff73-fe6f-4764-89f0-c1744ad4546a","Type":"ContainerDied","Data":"61154274d24043eccac022f5e2bd5401e2847e6a1c5b72cdb845e740d39479ba"}
Oct 30 13:02:10.759920 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:10.758612    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"]
Oct 30 13:02:10.774085 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:10.774052    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dea6ee-openshift-default-85794dd75-skxkx"]
Oct 30 13:02:10.877460 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:10.877437    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"
Oct 30 13:02:11.061873 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:11.061843    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff38ff73-fe6f-4764-89f0-c1744ad4546a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff38ff73-fe6f-4764-89f0-c1744ad4546a-maas-api-xks-ca\") pod \"ff38ff73-fe6f-4764-89f0-c1744ad4546a\" (UID: \"ff38ff73-fe6f-4764-89f0-c1744ad4546a\") "
Oct 30 13:02:11.062143 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:11.062124    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff38ff73-fe6f-4764-89f0-c1744ad4546a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff38ff73-fe6f-4764-89f0-c1744ad4546a-maas-api-openshift-ca\") pod \"ff38ff73-fe6f-4764-89f0-c1744ad4546a\" (UID: \"ff38ff73-fe6f-4764-89f0-c1744ad4546a\") "
Oct 30 13:02:11.062230 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:11.062216    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff38ff73-fe6f-4764-89f0-c1744ad4546a-kube-api-access-rrfcl\" (UniqueName: \"kubernetes.io/projected/ff38ff73-fe6f-4764-89f0-c1744ad4546a-kube-api-access-rrfcl\") pod \"ff38ff73-fe6f-4764-89f0-c1744ad4546a\" (UID: \"ff38ff73-fe6f-4764-89f0-c1744ad4546a\") "
Oct 30 13:02:11.062587 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:11.062556    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff38ff73-fe6f-4764-89f0-c1744ad4546a-maas-api-openshift-ca" pod "ff38ff73-fe6f-4764-89f0-c1744ad4546a" (UID: "ff38ff73-fe6f-4764-89f0-c1744ad4546a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:11.062823 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:11.062808    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff38ff73-fe6f-4764-89f0-c1744ad4546a-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:11.064184 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:11.064163    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff38ff73-fe6f-4764-89f0-c1744ad4546a-maas-api-xks-ca" pod "ff38ff73-fe6f-4764-89f0-c1744ad4546a" (UID: "ff38ff73-fe6f-4764-89f0-c1744ad4546a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:11.065277 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:11.065252    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff38ff73-fe6f-4764-89f0-c1744ad4546a-kube-api-access-rrfcl" pod "ff38ff73-fe6f-4764-89f0-c1744ad4546a" (UID: "ff38ff73-fe6f-4764-89f0-c1744ad4546a"). InnerVolumeSpecName "kube-api-access-rrfcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:11.164306 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:11.164271    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrfcl\" (UniqueName: \"kubernetes.io/projected/ff38ff73-fe6f-4764-89f0-c1744ad4546a-kube-api-access-rrfcl\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:11.164306 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:11.164300    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff38ff73-fe6f-4764-89f0-c1744ad4546a-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:11.619946 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:11.619914    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" path="/var/lib/kubelet/pods/1eb8b513-9fce-4ddd-9555-9a61cbe430dc/volumes"
Oct 30 13:02:11.629506 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:11.629476    5837 generic.go:356] "Generic (PLEG): container finished" podID="a2f13e3f-c6ea-48b1-b60f-84418d3f0447" containerID="58ff2d91238c0e6e49ede3337e42165fdbd43e4f07fade7782f866636fb1431d" exitCode=0
Oct 30 13:02:11.629814 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:11.629792    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5" event={"ID":"a2f13e3f-c6ea-48b1-b60f-84418d3f0447","Type":"ContainerDied","Data":"58ff2d91238c0e6e49ede3337e42165fdbd43e4f07fade7782f866636fb1431d"}
Oct 30 13:02:11.644343 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:11.644317    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x" event={"ID":"ff38ff73-fe6f-4764-89f0-c1744ad4546a","Type":"ContainerDied","Data":"568e015f0f4334b0f308303fa841907673becd1699d78d52fc5b7bfd352de807"}
Oct 30 13:02:11.644555 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:11.644538    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="568e015f0f4334b0f308303fa841907673becd1699d78d52fc5b7bfd352de807"
Oct 30 13:02:11.644555 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:11.644355    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"
Oct 30 13:02:12.947411 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:12.947364    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"
Oct 30 13:02:13.014948 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.013754    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"]
Oct 30 13:02:13.015831 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.015801    5837 cpu_manager.go:414] "containerMap: removing container" podUID="ff38ff73-fe6f-4764-89f0-c1744ad4546a" containerName="revoke-keys"
Oct 30 13:02:13.015831 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.015831    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff38ff73-fe6f-4764-89f0-c1744ad4546a" containerName="revoke-keys"
Oct 30 13:02:13.016038 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.015872    5837 cpu_manager.go:414] "containerMap: removing container" podUID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" containerName="istio-proxy"
Oct 30 13:02:13.016038 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.015880    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" containerName="istio-proxy"
Oct 30 13:02:13.016038 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.015906    5837 cpu_manager.go:414] "containerMap: removing container" podUID="a2f13e3f-c6ea-48b1-b60f-84418d3f0447" containerName="revoke-keys"
Oct 30 13:02:13.016038 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.015914    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f13e3f-c6ea-48b1-b60f-84418d3f0447" containerName="revoke-keys"
Oct 30 13:02:13.016241 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.016139    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff38ff73-fe6f-4764-89f0-c1744ad4546a" containerName="revoke-keys"
Oct 30 13:02:13.016241 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.016157    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" containerName="istio-proxy"
Oct 30 13:02:13.016241 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.016171    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f13e3f-c6ea-48b1-b60f-84418d3f0447" containerName="revoke-keys"
Oct 30 13:02:13.021242 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.021202    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.043236 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.043208    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-2382a9-openshift-default-dockercfg-xqndt\""
Oct 30 13:02:13.045242 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.045141    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"]
Oct 30 13:02:13.192896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.181830    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-maas-api-xks-ca\") pod \"a2f13e3f-c6ea-48b1-b60f-84418d3f0447\" (UID: \"a2f13e3f-c6ea-48b1-b60f-84418d3f0447\") "
Oct 30 13:02:13.192896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.182071    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-maas-api-openshift-ca\") pod \"a2f13e3f-c6ea-48b1-b60f-84418d3f0447\" (UID: \"a2f13e3f-c6ea-48b1-b60f-84418d3f0447\") "
Oct 30 13:02:13.192896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.182132    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-kube-api-access-69qkv\" (UniqueName: \"kubernetes.io/projected/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-kube-api-access-69qkv\") pod \"a2f13e3f-c6ea-48b1-b60f-84418d3f0447\" (UID: \"a2f13e3f-c6ea-48b1-b60f-84418d3f0447\") "
Oct 30 13:02:13.203762 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.203712    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-maas-api-openshift-ca" pod "a2f13e3f-c6ea-48b1-b60f-84418d3f0447" (UID: "a2f13e3f-c6ea-48b1-b60f-84418d3f0447"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:13.205135 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.205086    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-maas-api-xks-ca" pod "a2f13e3f-c6ea-48b1-b60f-84418d3f0447" (UID: "a2f13e3f-c6ea-48b1-b60f-84418d3f0447"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:13.208322 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.208281    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-kube-api-access-69qkv" pod "a2f13e3f-c6ea-48b1-b60f-84418d3f0447" (UID: "a2f13e3f-c6ea-48b1-b60f-84418d3f0447"). InnerVolumeSpecName "kube-api-access-69qkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:13.233306 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.232815    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-data\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.233306 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.232872    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-token\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.233306 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.232899    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istiod-ca-cert\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.233306 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.232955    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-workload-certs\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.233306 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.232988    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-workload-socket\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.233306 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.233024    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-podinfo\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.233306 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.233052    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-envoy\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.233306 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.233174    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-credential-socket\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.233306 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.233203    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t467b\" (UniqueName: \"kubernetes.io/projected/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-kube-api-access-t467b\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.233802 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.233358    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:13.233802 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.233374    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69qkv\" (UniqueName: \"kubernetes.io/projected/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-kube-api-access-69qkv\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:13.233802 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.233409    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f13e3f-c6ea-48b1-b60f-84418d3f0447-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:13.334672 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.334633    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-workload-socket\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.334884 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.334696    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-podinfo\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.334884 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.334729    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-envoy\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.334884 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.334848    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-credential-socket\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.334884 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.334876    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t467b\" (UniqueName: \"kubernetes.io/projected/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-kube-api-access-t467b\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.335111 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.335075    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-workload-socket\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.335169 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.335106    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-data\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.335169 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.335150    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-token\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.335271 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.335175    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istiod-ca-cert\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.335271 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.335220    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-credential-socket\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.335271 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.335231    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-workload-certs\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.335487 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.335464    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-data\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.335697 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.335675    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-workload-certs\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.335919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.335895    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istiod-ca-cert\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.341444 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.338486    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-envoy\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.341444 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.338555    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-podinfo\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.354726 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.354657    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-token\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.355371 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.355342    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t467b\" (UniqueName: \"kubernetes.io/projected/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-kube-api-access-t467b\") pod \"e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") " pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.362441 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.360901    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:13.664664 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.664633    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"
Oct 30 13:02:13.664820 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.664688    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5" event={"ID":"a2f13e3f-c6ea-48b1-b60f-84418d3f0447","Type":"ContainerDied","Data":"c572e7956ed64eb8976cdb2d16ddc04b452b562be354a78d7bef3e50026f01f8"}
Oct 30 13:02:13.664820 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.664757    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c572e7956ed64eb8976cdb2d16ddc04b452b562be354a78d7bef3e50026f01f8"
Oct 30 13:02:13.728173 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.728139    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"]
Oct 30 13:02:13.729727 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:02:13.729702    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2ab740_6bfa_438b_a878_e8a852a9b1a1.slice/crio-e2c2c311839860178bdd2703e61f76d615c4ccb7437420cfe2e8a4e59ff95712 WatchSource:0}: Error finding container e2c2c311839860178bdd2703e61f76d615c4ccb7437420cfe2e8a4e59ff95712: Status 404 returned error can't find the container with id e2c2c311839860178bdd2703e61f76d615c4ccb7437420cfe2e8a4e59ff95712
Oct 30 13:02:13.732671 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.732641    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:13.732743 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.732705    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:13.732743 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:13.732734    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:14.676153 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:14.676114    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" event={"ID":"9e2ab740-6bfa-438b-a878-e8a852a9b1a1","Type":"ContainerStarted","Data":"ff6b665d8257d32f953ec76d48bf420e6fff2572de455306840fc3cb4bed198a"}
Oct 30 13:02:14.676696 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:14.676219    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" event={"ID":"9e2ab740-6bfa-438b-a878-e8a852a9b1a1","Type":"ContainerStarted","Data":"e2c2c311839860178bdd2703e61f76d615c4ccb7437420cfe2e8a4e59ff95712"}
Oct 30 13:02:14.751428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:14.751289    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" podStartSLOduration=2.751267387 podStartE2EDuration="2.751267387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:14.735212822 +0000 UTC m=+1523.259497040" watchObservedRunningTime="2026-10-30 13:02:14.751267387 +0000 UTC m=+1523.275551607"
Oct 30 13:02:14.883071 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:14.882687    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:14.883071 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:14.882753    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92" podUID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:15.100610 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:02:15.100575    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:02:15.361509 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:15.361427    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:15.367843 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:15.367811    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:15.686308 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:15.686219    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:15.687175 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:15.687147    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:02:16.545415 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:16.545367    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"]
Oct 30 13:02:18.750713 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:18.750682    5837 generic.go:356] "Generic (PLEG): container finished" podID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerID="cc2a500ec16ebe534e04419d11c56dd3f5eb9f695c2782752a8f4390b4ca697f" exitCode=0
Oct 30 13:02:18.751143 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:18.750748    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92" event={"ID":"cee173cf-090c-42b7-8427-36e58cc4f3ba","Type":"ContainerDied","Data":"cc2a500ec16ebe534e04419d11c56dd3f5eb9f695c2782752a8f4390b4ca697f"}
Oct 30 13:02:18.751190 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:18.751168    5837 scope.go:122] "RemoveContainer" containerID="cc2a500ec16ebe534e04419d11c56dd3f5eb9f695c2782752a8f4390b4ca697f"
Oct 30 13:02:18.752977 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:18.752944    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:18.753101 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:18.753023    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:18.753101 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:18.753055    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:19.477347 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.477315    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"]
Oct 30 13:02:19.478002 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.477987    5837 cpu_manager.go:414] "containerMap: removing container" podUID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" containerName="istio-proxy"
Oct 30 13:02:19.478081 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.478003    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" containerName="istio-proxy"
Oct 30 13:02:19.478156 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.478137    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb8b513-9fce-4ddd-9555-9a61cbe430dc" containerName="istio-proxy"
Oct 30 13:02:19.481403 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.481367    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"
Oct 30 13:02:19.511420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.508736    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-2382a9\""
Oct 30 13:02:19.511420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.508884    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-2382a9-dockercfg-gtwrt\""
Oct 30 13:02:19.516670 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.516637    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"]
Oct 30 13:02:19.581808 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.581776    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"]
Oct 30 13:02:19.629422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.627375    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ebbba38-3e78-485f-abc4-fda302f2888e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd\" (UID: \"3ebbba38-3e78-485f-abc4-fda302f2888e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"
Oct 30 13:02:19.629422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.627526    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mprng\" (UniqueName: \"kubernetes.io/projected/3ebbba38-3e78-485f-abc4-fda302f2888e-kube-api-access-mprng\") pod \"payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd\" (UID: \"3ebbba38-3e78-485f-abc4-fda302f2888e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"
Oct 30 13:02:19.730605 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.729793    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ebbba38-3e78-485f-abc4-fda302f2888e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd\" (UID: \"3ebbba38-3e78-485f-abc4-fda302f2888e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"
Oct 30 13:02:19.730605 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.729928    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mprng\" (UniqueName: \"kubernetes.io/projected/3ebbba38-3e78-485f-abc4-fda302f2888e-kube-api-access-mprng\") pod \"payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd\" (UID: \"3ebbba38-3e78-485f-abc4-fda302f2888e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"
Oct 30 13:02:19.751461 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.731775    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ebbba38-3e78-485f-abc4-fda302f2888e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd\" (UID: \"3ebbba38-3e78-485f-abc4-fda302f2888e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"
Oct 30 13:02:19.868421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.847022    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mprng\" (UniqueName: \"kubernetes.io/projected/3ebbba38-3e78-485f-abc4-fda302f2888e-kube-api-access-mprng\") pod \"payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd\" (UID: \"3ebbba38-3e78-485f-abc4-fda302f2888e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"
Oct 30 13:02:19.898860 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.897481    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"
Oct 30 13:02:19.943244 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.943200    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34067f6-150e-46ca-b023-912b4c89c124-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx\" (UID: \"a34067f6-150e-46ca-b023-912b4c89c124\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"
Oct 30 13:02:19.943560 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.943538    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8cth\" (UniqueName: \"kubernetes.io/projected/a34067f6-150e-46ca-b023-912b4c89c124-kube-api-access-f8cth\") pod \"payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx\" (UID: \"a34067f6-150e-46ca-b023-912b4c89c124\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"
Oct 30 13:02:19.956928 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.956903    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"]
Oct 30 13:02:19.958952 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:19.958927    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92" event={"ID":"cee173cf-090c-42b7-8427-36e58cc4f3ba","Type":"ContainerStarted","Data":"ff087d509b01f5a47111f54afb25632beef37f689ff620ddc7dd936c34a5ad4f"}
Oct 30 13:02:20.044885 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.044787    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8cth\" (UniqueName: \"kubernetes.io/projected/a34067f6-150e-46ca-b023-912b4c89c124-kube-api-access-f8cth\") pod \"payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx\" (UID: \"a34067f6-150e-46ca-b023-912b4c89c124\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"
Oct 30 13:02:20.045077 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.044954    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34067f6-150e-46ca-b023-912b4c89c124-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx\" (UID: \"a34067f6-150e-46ca-b023-912b4c89c124\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"
Oct 30 13:02:20.045659 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.045640    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34067f6-150e-46ca-b023-912b4c89c124-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx\" (UID: \"a34067f6-150e-46ca-b023-912b4c89c124\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"
Oct 30 13:02:20.066482 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.066451    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8cth\" (UniqueName: \"kubernetes.io/projected/a34067f6-150e-46ca-b023-912b4c89c124-kube-api-access-f8cth\") pod \"payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx\" (UID: \"a34067f6-150e-46ca-b023-912b4c89c124\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"
Oct 30 13:02:20.134003 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.133971    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"
Oct 30 13:02:20.258751 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.258711    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"
Oct 30 13:02:20.462905 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.462880    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"]
Oct 30 13:02:20.463686 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:02:20.463662    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebbba38_3e78_485f_abc4_fda302f2888e.slice/crio-bd587c620ffb694b5c405a2ee857a972fbb08ce0510682f3dec2d7c117388da0 WatchSource:0}: Error finding container bd587c620ffb694b5c405a2ee857a972fbb08ce0510682f3dec2d7c117388da0: Status 404 returned error can't find the container with id bd587c620ffb694b5c405a2ee857a972fbb08ce0510682f3dec2d7c117388da0
Oct 30 13:02:20.615008 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:02:20.614973    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34067f6_150e_46ca_b023_912b4c89c124.slice/crio-91971036180c34eb87ae043a967691ba5a4e30bf323ade72457fa8ca5ea5dfd2 WatchSource:0}: Error finding container 91971036180c34eb87ae043a967691ba5a4e30bf323ade72457fa8ca5ea5dfd2: Status 404 returned error can't find the container with id 91971036180c34eb87ae043a967691ba5a4e30bf323ade72457fa8ca5ea5dfd2
Oct 30 13:02:20.616939 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.616910    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"]
Oct 30 13:02:20.682441 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.682385    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:02:20.682576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.682559    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:02:20.682632 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.682603    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92" podUID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 13:02:20.968269 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.968181    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd" event={"ID":"3ebbba38-3e78-485f-abc4-fda302f2888e","Type":"ContainerStarted","Data":"a6674a10a8a91507cfc773111b7b2332032a1e2b6a6f1567a849bb26e8f0e516"}
Oct 30 13:02:20.968269 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.968235    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd" event={"ID":"3ebbba38-3e78-485f-abc4-fda302f2888e","Type":"ContainerStarted","Data":"bd587c620ffb694b5c405a2ee857a972fbb08ce0510682f3dec2d7c117388da0"}
Oct 30 13:02:20.968827 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.968370    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"
Oct 30 13:02:20.970120 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.970091    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx" event={"ID":"a34067f6-150e-46ca-b023-912b4c89c124","Type":"ContainerStarted","Data":"88dd3b734b33ef440e356af69711889498e96f0f50951c68d0dfd88e8fd26a1a"}
Oct 30 13:02:20.970284 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.970269    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"
Oct 30 13:02:20.970347 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:20.970295    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx" event={"ID":"a34067f6-150e-46ca-b023-912b4c89c124","Type":"ContainerStarted","Data":"91971036180c34eb87ae043a967691ba5a4e30bf323ade72457fa8ca5ea5dfd2"}
Oct 30 13:02:21.014345 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.014283    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd" podStartSLOduration=2.014269595 podStartE2EDuration="2.014269595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:21.009321007 +0000 UTC m=+1529.533605226" watchObservedRunningTime="2026-10-30 13:02:21.014269595 +0000 UTC m=+1529.538553813"
Oct 30 13:02:21.049195 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.049147    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx" podStartSLOduration=2.049131787 podStartE2EDuration="2.049131787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:21.048263717 +0000 UTC m=+1529.572547935" watchObservedRunningTime="2026-10-30 13:02:21.049131787 +0000 UTC m=+1529.573416006"
Oct 30 13:02:21.245637 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.245606    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"]
Oct 30 13:02:21.249187 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.249169    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.268377 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.268344    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-096cc3-openshift-default-dockercfg-5rpdm\""
Oct 30 13:02:21.277503 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.277476    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"]
Oct 30 13:02:21.365449 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.365418    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-workload-certs\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.365638 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.365478    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hbxc\" (UniqueName: \"kubernetes.io/projected/7543408a-97b7-4a86-bfaf-9834106505bf-kube-api-access-9hbxc\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.365638 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.365624    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-workload-socket\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.365751 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.365659    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7543408a-97b7-4a86-bfaf-9834106505bf-istio-podinfo\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.365751 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.365739    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7543408a-97b7-4a86-bfaf-9834106505bf-istiod-ca-cert\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.365841 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.365764    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-istio-envoy\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.365841 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.365810    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-credential-socket\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.365957 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.365941    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-istio-data\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.365995 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.365988    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7543408a-97b7-4a86-bfaf-9834106505bf-istio-token\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.467595 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.467554    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7543408a-97b7-4a86-bfaf-9834106505bf-istio-token\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.467595 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.467600    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-workload-certs\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.467853 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.467641    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hbxc\" (UniqueName: \"kubernetes.io/projected/7543408a-97b7-4a86-bfaf-9834106505bf-kube-api-access-9hbxc\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.467853 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.467712    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-workload-socket\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.467853 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.467738    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7543408a-97b7-4a86-bfaf-9834106505bf-istio-podinfo\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.467853 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.467777    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7543408a-97b7-4a86-bfaf-9834106505bf-istiod-ca-cert\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.468093 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.467902    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-istio-envoy\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.468093 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.467961    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-credential-socket\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.468093 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.468061    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-istio-data\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.468226 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.468114    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-workload-certs\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.468226 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.468156    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-workload-socket\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.468447 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.468380    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-credential-socket\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.468546 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.468484    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-istio-data\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.468689 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.468646    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7543408a-97b7-4a86-bfaf-9834106505bf-istiod-ca-cert\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.470479 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.470454    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-istio-envoy\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.470607 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.470564    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7543408a-97b7-4a86-bfaf-9834106505bf-istio-podinfo\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.511490 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.511376    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hbxc\" (UniqueName: \"kubernetes.io/projected/7543408a-97b7-4a86-bfaf-9834106505bf-kube-api-access-9hbxc\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.512193 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.512168    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7543408a-97b7-4a86-bfaf-9834106505bf-istio-token\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.546386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.546353    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5" podUID="f03a6158-3836-4f91-b4b7-98ea98c2fb90" containerName="payload-pre-processing" containerID="cri-o://09fc1b0a29696cafa053bb787942ad53707260e6d716c65c9894fe27d318c156" gracePeriod=25
Oct 30 13:02:21.571260 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.571234    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:21.682291 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.682259    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:02:21.682601 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.682308    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92" podUID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 13:02:21.912351 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.912304    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"]
Oct 30 13:02:21.916625 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:02:21.916600    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7543408a_97b7_4a86_bfaf_9834106505bf.slice/crio-2bb1044c225434063b9e121aec17daaa4f2d4124cf6b522326f35be5dcbb476c WatchSource:0}: Error finding container 2bb1044c225434063b9e121aec17daaa4f2d4124cf6b522326f35be5dcbb476c: Status 404 returned error can't find the container with id 2bb1044c225434063b9e121aec17daaa4f2d4124cf6b522326f35be5dcbb476c
Oct 30 13:02:21.917482 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.917459    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"
Oct 30 13:02:21.919488 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.919456    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:21.919557 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.919525    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:21.919603 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.919556    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:21.976072 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.976043    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f03a6158-3836-4f91-b4b7-98ea98c2fb90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03a6158-3836-4f91-b4b7-98ea98c2fb90-plugins-config-volume\") pod \"f03a6158-3836-4f91-b4b7-98ea98c2fb90\" (UID: \"f03a6158-3836-4f91-b4b7-98ea98c2fb90\") "
Oct 30 13:02:21.976452 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.976330    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f03a6158-3836-4f91-b4b7-98ea98c2fb90-kube-api-access-mtzmn\" (UniqueName: \"kubernetes.io/projected/f03a6158-3836-4f91-b4b7-98ea98c2fb90-kube-api-access-mtzmn\") pod \"f03a6158-3836-4f91-b4b7-98ea98c2fb90\" (UID: \"f03a6158-3836-4f91-b4b7-98ea98c2fb90\") "
Oct 30 13:02:21.976868 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.976834    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03a6158-3836-4f91-b4b7-98ea98c2fb90-plugins-config-volume" pod "f03a6158-3836-4f91-b4b7-98ea98c2fb90" (UID: "f03a6158-3836-4f91-b4b7-98ea98c2fb90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:21.977101 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.977083    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03a6158-3836-4f91-b4b7-98ea98c2fb90-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:21.995825 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:21.981875    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03a6158-3836-4f91-b4b7-98ea98c2fb90-kube-api-access-mtzmn" pod "f03a6158-3836-4f91-b4b7-98ea98c2fb90" (UID: "f03a6158-3836-4f91-b4b7-98ea98c2fb90"). InnerVolumeSpecName "kube-api-access-mtzmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:22.029031 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.028998    5837 generic.go:356] "Generic (PLEG): container finished" podID="f03a6158-3836-4f91-b4b7-98ea98c2fb90" containerID="09fc1b0a29696cafa053bb787942ad53707260e6d716c65c9894fe27d318c156" exitCode=0
Oct 30 13:02:22.029177 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.029124    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5" event={"ID":"f03a6158-3836-4f91-b4b7-98ea98c2fb90","Type":"ContainerDied","Data":"09fc1b0a29696cafa053bb787942ad53707260e6d716c65c9894fe27d318c156"}
Oct 30 13:02:22.029249 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.029184    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5" event={"ID":"f03a6158-3836-4f91-b4b7-98ea98c2fb90","Type":"ContainerDied","Data":"29c9cf3698caea17d47e4b2c28f34b2cdbf67943e671308d2f26e14d0794f187"}
Oct 30 13:02:22.029249 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.029203    5837 scope.go:122] "RemoveContainer" containerID="09fc1b0a29696cafa053bb787942ad53707260e6d716c65c9894fe27d318c156"
Oct 30 13:02:22.029350 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.029322    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"
Oct 30 13:02:22.031364 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.031333    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8" event={"ID":"7543408a-97b7-4a86-bfaf-9834106505bf","Type":"ContainerStarted","Data":"2bb1044c225434063b9e121aec17daaa4f2d4124cf6b522326f35be5dcbb476c"}
Oct 30 13:02:22.041941 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.041440    5837 scope.go:122] "RemoveContainer" containerID="09fc1b0a29696cafa053bb787942ad53707260e6d716c65c9894fe27d318c156"
Oct 30 13:02:22.041941 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:02:22.041704    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09fc1b0a29696cafa053bb787942ad53707260e6d716c65c9894fe27d318c156\": container with ID starting with 09fc1b0a29696cafa053bb787942ad53707260e6d716c65c9894fe27d318c156 not found: ID does not exist" containerID="09fc1b0a29696cafa053bb787942ad53707260e6d716c65c9894fe27d318c156"
Oct 30 13:02:22.041941 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.041741    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09fc1b0a29696cafa053bb787942ad53707260e6d716c65c9894fe27d318c156"} err="failed to get container status \"09fc1b0a29696cafa053bb787942ad53707260e6d716c65c9894fe27d318c156\": rpc error: code = NotFound desc = could not find container \"09fc1b0a29696cafa053bb787942ad53707260e6d716c65c9894fe27d318c156\": container with ID starting with 09fc1b0a29696cafa053bb787942ad53707260e6d716c65c9894fe27d318c156 not found: ID does not exist"
Oct 30 13:02:22.079378 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.079331    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtzmn\" (UniqueName: \"kubernetes.io/projected/f03a6158-3836-4f91-b4b7-98ea98c2fb90-kube-api-access-mtzmn\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:22.091423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.085268    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"]
Oct 30 13:02:22.114334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.114253    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-5c77644c9-7w5s5"]
Oct 30 13:02:22.682349 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.682310    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:02:22.682546 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.682374    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92" podUID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 13:02:22.712470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.711946    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"]
Oct 30 13:02:22.712470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.712187    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp" podUID="d17210fa-f13e-4df6-b6c6-f2f6cbc338e3" containerName="payload-processing" containerID="cri-o://b8fc9f00f0c0ba3cb3f1c17503e55c1030385efd0f717acaaf901d3626b148c9" gracePeriod=30
Oct 30 13:02:22.725615 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.725586    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"]
Oct 30 13:02:22.725821 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.725792    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2" podUID="4d021c3e-5871-4c5a-aa65-0244db8501de" containerName="payload-pre-processing" containerID="cri-o://4dd69895117c9bdad7613beafee2e1c5a37b70de9463e564c8f4df2c5fbc5c89" gracePeriod=30
Oct 30 13:02:22.841778 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:22.833027    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp" podUID="d17210fa-f13e-4df6-b6c6-f2f6cbc338e3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.169:9004: connect: connection refused"
Oct 30 13:02:23.562755 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:23.561483    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"]
Oct 30 13:02:23.562755 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:23.561716    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6" podUID="431d0ac1-5825-40c3-af3b-031fa5726ffa" containerName="payload-processing" containerID="cri-o://07cf213b6b60177226f9dc16983a9fda3b5592d98e5f2c6f550d2e56dd6fdd2c" gracePeriod=30
Oct 30 13:02:23.667691 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:23.667659    5837 generic.go:356] "Generic (PLEG): container finished" podID="d17210fa-f13e-4df6-b6c6-f2f6cbc338e3" containerID="b8fc9f00f0c0ba3cb3f1c17503e55c1030385efd0f717acaaf901d3626b148c9" exitCode=0
Oct 30 13:02:23.670881 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:23.670553    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f03a6158-3836-4f91-b4b7-98ea98c2fb90" path="/var/lib/kubelet/pods/f03a6158-3836-4f91-b4b7-98ea98c2fb90/volumes"
Oct 30 13:02:23.672163 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:23.671991    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp" event={"ID":"d17210fa-f13e-4df6-b6c6-f2f6cbc338e3","Type":"ContainerDied","Data":"b8fc9f00f0c0ba3cb3f1c17503e55c1030385efd0f717acaaf901d3626b148c9"}
Oct 30 13:02:23.672163 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:23.672045    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8" event={"ID":"7543408a-97b7-4a86-bfaf-9834106505bf","Type":"ContainerStarted","Data":"c6fddaa6d412f069419f505ae7b6a707955cdef8a8f03aec8e507917ee6b7f6a"}
Oct 30 13:02:23.675899 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:23.675870    5837 generic.go:356] "Generic (PLEG): container finished" podID="4d021c3e-5871-4c5a-aa65-0244db8501de" containerID="4dd69895117c9bdad7613beafee2e1c5a37b70de9463e564c8f4df2c5fbc5c89" exitCode=0
Oct 30 13:02:23.676005 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:23.675939    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2" event={"ID":"4d021c3e-5871-4c5a-aa65-0244db8501de","Type":"ContainerDied","Data":"4dd69895117c9bdad7613beafee2e1c5a37b70de9463e564c8f4df2c5fbc5c89"}
Oct 30 13:02:23.682984 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:23.682960    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:02:23.683171 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:23.683146    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92" podUID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 13:02:23.754743 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:23.754623    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"]
Oct 30 13:02:23.762547 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:23.762519    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-096cc3-h9s9x"]
Oct 30 13:02:24.254419 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.251058    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"
Oct 30 13:02:24.254419 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.253604    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"
Oct 30 13:02:24.317420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.316409    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17210fa-f13e-4df6-b6c6-f2f6cbc338e3-kube-api-access-sck54\" (UniqueName: \"kubernetes.io/projected/d17210fa-f13e-4df6-b6c6-f2f6cbc338e3-kube-api-access-sck54\") pod \"d17210fa-f13e-4df6-b6c6-f2f6cbc338e3\" (UID: \"d17210fa-f13e-4df6-b6c6-f2f6cbc338e3\") "
Oct 30 13:02:24.317420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.316558    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d021c3e-5871-4c5a-aa65-0244db8501de-kube-api-access-bhbzh\" (UniqueName: \"kubernetes.io/projected/4d021c3e-5871-4c5a-aa65-0244db8501de-kube-api-access-bhbzh\") pod \"4d021c3e-5871-4c5a-aa65-0244db8501de\" (UID: \"4d021c3e-5871-4c5a-aa65-0244db8501de\") "
Oct 30 13:02:24.317420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.316654    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d021c3e-5871-4c5a-aa65-0244db8501de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d021c3e-5871-4c5a-aa65-0244db8501de-plugins-config-volume\") pod \"4d021c3e-5871-4c5a-aa65-0244db8501de\" (UID: \"4d021c3e-5871-4c5a-aa65-0244db8501de\") "
Oct 30 13:02:24.317420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.316750    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d17210fa-f13e-4df6-b6c6-f2f6cbc338e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17210fa-f13e-4df6-b6c6-f2f6cbc338e3-plugins-config-volume\") pod \"d17210fa-f13e-4df6-b6c6-f2f6cbc338e3\" (UID: \"d17210fa-f13e-4df6-b6c6-f2f6cbc338e3\") "
Oct 30 13:02:24.330555 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.329123    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8" podStartSLOduration=3.329105676 podStartE2EDuration="3.329105676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:23.815900833 +0000 UTC m=+1532.340185054" watchObservedRunningTime="2026-10-30 13:02:24.329105676 +0000 UTC m=+1532.853389898"
Oct 30 13:02:24.344714 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.338474    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17210fa-f13e-4df6-b6c6-f2f6cbc338e3-plugins-config-volume" pod "d17210fa-f13e-4df6-b6c6-f2f6cbc338e3" (UID: "d17210fa-f13e-4df6-b6c6-f2f6cbc338e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:24.344714 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.343127    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d021c3e-5871-4c5a-aa65-0244db8501de-plugins-config-volume" pod "4d021c3e-5871-4c5a-aa65-0244db8501de" (UID: "4d021c3e-5871-4c5a-aa65-0244db8501de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:24.344714 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.344578    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d021c3e-5871-4c5a-aa65-0244db8501de-kube-api-access-bhbzh" pod "4d021c3e-5871-4c5a-aa65-0244db8501de" (UID: "4d021c3e-5871-4c5a-aa65-0244db8501de"). InnerVolumeSpecName "kube-api-access-bhbzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:24.346446 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.345532    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17210fa-f13e-4df6-b6c6-f2f6cbc338e3-kube-api-access-sck54" pod "d17210fa-f13e-4df6-b6c6-f2f6cbc338e3" (UID: "d17210fa-f13e-4df6-b6c6-f2f6cbc338e3"). InnerVolumeSpecName "kube-api-access-sck54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:24.419471 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.418704    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sck54\" (UniqueName: \"kubernetes.io/projected/d17210fa-f13e-4df6-b6c6-f2f6cbc338e3-kube-api-access-sck54\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:24.419471 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.418749    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhbzh\" (UniqueName: \"kubernetes.io/projected/4d021c3e-5871-4c5a-aa65-0244db8501de-kube-api-access-bhbzh\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:24.419471 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.418765    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d021c3e-5871-4c5a-aa65-0244db8501de-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:24.419471 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.418779    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17210fa-f13e-4df6-b6c6-f2f6cbc338e3-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:24.570161 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.570136    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"
Oct 30 13:02:24.571662 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.571644    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:24.572465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.572446    5837 patch_prober.go:29] interesting pod/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 13:02:24.572519 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.572491    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8" podUID="7543408a-97b7-4a86-bfaf-9834106505bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 13:02:24.700287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.700091    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:02:24.700287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.700146    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92" podUID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 13:02:24.703655 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.702888    5837 generic.go:356] "Generic (PLEG): container finished" podID="431d0ac1-5825-40c3-af3b-031fa5726ffa" containerID="07cf213b6b60177226f9dc16983a9fda3b5592d98e5f2c6f550d2e56dd6fdd2c" exitCode=0
Oct 30 13:02:24.703655 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.703024    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6" event={"ID":"431d0ac1-5825-40c3-af3b-031fa5726ffa","Type":"ContainerDied","Data":"07cf213b6b60177226f9dc16983a9fda3b5592d98e5f2c6f550d2e56dd6fdd2c"}
Oct 30 13:02:24.703655 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.703074    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6" event={"ID":"431d0ac1-5825-40c3-af3b-031fa5726ffa","Type":"ContainerDied","Data":"3345911042a263fb5b98edf2af12cc064c5f76734df76697e73aa224d3c15d4e"}
Oct 30 13:02:24.703655 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.703106    5837 scope.go:122] "RemoveContainer" containerID="07cf213b6b60177226f9dc16983a9fda3b5592d98e5f2c6f550d2e56dd6fdd2c"
Oct 30 13:02:24.703655 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.703209    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"
Oct 30 13:02:24.713420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.709563    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp" event={"ID":"d17210fa-f13e-4df6-b6c6-f2f6cbc338e3","Type":"ContainerDied","Data":"283f295628543d695d8b58c8cca41ffc9a5e5714c3708cf6960a749363a8f995"}
Oct 30 13:02:24.713420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.709667    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"
Oct 30 13:02:24.846002 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.822982    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/431d0ac1-5825-40c3-af3b-031fa5726ffa-kube-api-access-685tq\" (UniqueName: \"kubernetes.io/projected/431d0ac1-5825-40c3-af3b-031fa5726ffa-kube-api-access-685tq\") pod \"431d0ac1-5825-40c3-af3b-031fa5726ffa\" (UID: \"431d0ac1-5825-40c3-af3b-031fa5726ffa\") "
Oct 30 13:02:24.846002 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.823263    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/431d0ac1-5825-40c3-af3b-031fa5726ffa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/431d0ac1-5825-40c3-af3b-031fa5726ffa-plugins-config-volume\") pod \"431d0ac1-5825-40c3-af3b-031fa5726ffa\" (UID: \"431d0ac1-5825-40c3-af3b-031fa5726ffa\") "
Oct 30 13:02:24.846002 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.838710    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431d0ac1-5825-40c3-af3b-031fa5726ffa-plugins-config-volume" pod "431d0ac1-5825-40c3-af3b-031fa5726ffa" (UID: "431d0ac1-5825-40c3-af3b-031fa5726ffa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:24.865855 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.852249    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2" podUID="4d021c3e-5871-4c5a-aa65-0244db8501de" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.168:9004: i/o timeout"
Oct 30 13:02:24.882433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.873562    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"
Oct 30 13:02:24.891712 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.884707    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2" event={"ID":"4d021c3e-5871-4c5a-aa65-0244db8501de","Type":"ContainerDied","Data":"95d17bb336675c070d104e0d714bfff92ddcc53fcaa128a5c27e41954271c5f5"}
Oct 30 13:02:24.907697 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.907666    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"]
Oct 30 13:02:24.907914 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.907901    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e74c27-7nhdn"]
Oct 30 13:02:24.950455 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.923651    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431d0ac1-5825-40c3-af3b-031fa5726ffa-kube-api-access-685tq" pod "431d0ac1-5825-40c3-af3b-031fa5726ffa" (UID: "431d0ac1-5825-40c3-af3b-031fa5726ffa"). InnerVolumeSpecName "kube-api-access-685tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:24.950455 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.949994    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-685tq\" (UniqueName: \"kubernetes.io/projected/431d0ac1-5825-40c3-af3b-031fa5726ffa-kube-api-access-685tq\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:24.950455 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.950020    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/431d0ac1-5825-40c3-af3b-031fa5726ffa-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:24.965017 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.964666    5837 scope.go:122] "RemoveContainer" containerID="07cf213b6b60177226f9dc16983a9fda3b5592d98e5f2c6f550d2e56dd6fdd2c"
Oct 30 13:02:24.966612 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:02:24.965505    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07cf213b6b60177226f9dc16983a9fda3b5592d98e5f2c6f550d2e56dd6fdd2c\": container with ID starting with 07cf213b6b60177226f9dc16983a9fda3b5592d98e5f2c6f550d2e56dd6fdd2c not found: ID does not exist" containerID="07cf213b6b60177226f9dc16983a9fda3b5592d98e5f2c6f550d2e56dd6fdd2c"
Oct 30 13:02:24.966612 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.965556    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07cf213b6b60177226f9dc16983a9fda3b5592d98e5f2c6f550d2e56dd6fdd2c"} err="failed to get container status \"07cf213b6b60177226f9dc16983a9fda3b5592d98e5f2c6f550d2e56dd6fdd2c\": rpc error: code = NotFound desc = could not find container \"07cf213b6b60177226f9dc16983a9fda3b5592d98e5f2c6f550d2e56dd6fdd2c\": container with ID starting with 07cf213b6b60177226f9dc16983a9fda3b5592d98e5f2c6f550d2e56dd6fdd2c not found: ID does not exist"
Oct 30 13:02:24.966612 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.965580    5837 scope.go:122] "RemoveContainer" containerID="b8fc9f00f0c0ba3cb3f1c17503e55c1030385efd0f717acaaf901d3626b148c9"
Oct 30 13:02:24.972380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.972349    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"]
Oct 30 13:02:24.979500 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:24.979477    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096cc3-68885f5d96-p2smp"]
Oct 30 13:02:25.066836 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:25.062311    5837 scope.go:122] "RemoveContainer" containerID="4dd69895117c9bdad7613beafee2e1c5a37b70de9463e564c8f4df2c5fbc5c89"
Oct 30 13:02:25.069713 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:25.069680    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"]
Oct 30 13:02:25.080697 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:25.080672    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096cc3-7d5bfb9cc9-gb5n2"]
Oct 30 13:02:25.143689 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:25.143596    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"]
Oct 30 13:02:25.148303 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:25.148275    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e74c27-6776bdb4fb-bwxs6"]
Oct 30 13:02:25.572551 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:25.572518    5837 patch_prober.go:29] interesting pod/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 13:02:25.572983 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:25.572573    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8" podUID="7543408a-97b7-4a86-bfaf-9834106505bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 13:02:25.630186 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:25.630153    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9ad268-8f35-41d0-92e2-c12fe92f6db0" path="/var/lib/kubelet/pods/0f9ad268-8f35-41d0-92e2-c12fe92f6db0/volumes"
Oct 30 13:02:25.630643 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:25.630626    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="431d0ac1-5825-40c3-af3b-031fa5726ffa" path="/var/lib/kubelet/pods/431d0ac1-5825-40c3-af3b-031fa5726ffa/volumes"
Oct 30 13:02:25.630992 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:25.630980    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d021c3e-5871-4c5a-aa65-0244db8501de" path="/var/lib/kubelet/pods/4d021c3e-5871-4c5a-aa65-0244db8501de/volumes"
Oct 30 13:02:25.631679 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:25.631663    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d17210fa-f13e-4df6-b6c6-f2f6cbc338e3" path="/var/lib/kubelet/pods/d17210fa-f13e-4df6-b6c6-f2f6cbc338e3/volumes"
Oct 30 13:02:25.632090 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:25.632076    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff38ff73-fe6f-4764-89f0-c1744ad4546a" path="/var/lib/kubelet/pods/ff38ff73-fe6f-4764-89f0-c1744ad4546a/volumes"
Oct 30 13:02:25.682229 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:25.682185    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:02:25.682523 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:25.682250    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92" podUID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 13:02:26.572032 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:26.571996    5837 patch_prober.go:29] interesting pod/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 13:02:26.572204 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:26.572050    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8" podUID="7543408a-97b7-4a86-bfaf-9834106505bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 13:02:26.709441 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:26.705729    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:02:26.709441 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:26.707185    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:02:26.710085 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:26.710061    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:02:27.571860 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:27.571827    5837 patch_prober.go:29] interesting pod/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 13:02:27.572015 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:27.571875    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8" podUID="7543408a-97b7-4a86-bfaf-9834106505bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 13:02:28.571897 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:28.571863    5837 patch_prober.go:29] interesting pod/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 13:02:28.572290 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:28.571913    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8" podUID="7543408a-97b7-4a86-bfaf-9834106505bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 13:02:29.576062 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:29.576034    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:29.576547 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:29.576530    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:29.577283 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:29.577268    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:31.183093 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:31.181806    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"]
Oct 30 13:02:32.035004 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.034931    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"
Oct 30 13:02:32.035004 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.034992    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"
Oct 30 13:02:32.035537 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.035500    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8" podUID="7543408a-97b7-4a86-bfaf-9834106505bf" containerName="istio-proxy" containerID="cri-o://c6fddaa6d412f069419f505ae7b6a707955cdef8a8f03aec8e507917ee6b7f6a" gracePeriod=30
Oct 30 13:02:32.421776 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.421669    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"]
Oct 30 13:02:32.427458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.422762    5837 cpu_manager.go:414] "containerMap: removing container" podUID="d17210fa-f13e-4df6-b6c6-f2f6cbc338e3" containerName="payload-processing"
Oct 30 13:02:32.427458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.422786    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17210fa-f13e-4df6-b6c6-f2f6cbc338e3" containerName="payload-processing"
Oct 30 13:02:32.427458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.422811    5837 cpu_manager.go:414] "containerMap: removing container" podUID="4d021c3e-5871-4c5a-aa65-0244db8501de" containerName="payload-pre-processing"
Oct 30 13:02:32.427458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.422818    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d021c3e-5871-4c5a-aa65-0244db8501de" containerName="payload-pre-processing"
Oct 30 13:02:32.427458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.422864    5837 cpu_manager.go:414] "containerMap: removing container" podUID="431d0ac1-5825-40c3-af3b-031fa5726ffa" containerName="payload-processing"
Oct 30 13:02:32.427458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.422871    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431d0ac1-5825-40c3-af3b-031fa5726ffa" containerName="payload-processing"
Oct 30 13:02:32.427458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.422886    5837 cpu_manager.go:414] "containerMap: removing container" podUID="f03a6158-3836-4f91-b4b7-98ea98c2fb90" containerName="payload-pre-processing"
Oct 30 13:02:32.427458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.422893    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03a6158-3836-4f91-b4b7-98ea98c2fb90" containerName="payload-pre-processing"
Oct 30 13:02:32.427458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.423059    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17210fa-f13e-4df6-b6c6-f2f6cbc338e3" containerName="payload-processing"
Oct 30 13:02:32.427458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.423075    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d021c3e-5871-4c5a-aa65-0244db8501de" containerName="payload-pre-processing"
Oct 30 13:02:32.427458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.423093    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="431d0ac1-5825-40c3-af3b-031fa5726ffa" containerName="payload-processing"
Oct 30 13:02:32.427458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.423104    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f03a6158-3836-4f91-b4b7-98ea98c2fb90" containerName="payload-pre-processing"
Oct 30 13:02:32.429873 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.429839    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"
Oct 30 13:02:32.432501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.432155    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"]
Oct 30 13:02:32.451083 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.451058    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a81a48\""
Oct 30 13:02:32.451224 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.451121    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a81a48-dockercfg-wr74s\""
Oct 30 13:02:32.487497 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.487459    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542ca25a-e199-4923-bd3f-b39860cb678c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5\" (UID: \"542ca25a-e199-4923-bd3f-b39860cb678c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"
Oct 30 13:02:32.487657 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.487512    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wbp8\" (UniqueName: \"kubernetes.io/projected/542ca25a-e199-4923-bd3f-b39860cb678c-kube-api-access-5wbp8\") pod \"payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5\" (UID: \"542ca25a-e199-4923-bd3f-b39860cb678c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"
Oct 30 13:02:32.562276 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.562246    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"]
Oct 30 13:02:32.565893 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.565869    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"
Oct 30 13:02:32.589142 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.589110    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"]
Oct 30 13:02:32.590164 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.590133    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542ca25a-e199-4923-bd3f-b39860cb678c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5\" (UID: \"542ca25a-e199-4923-bd3f-b39860cb678c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"
Oct 30 13:02:32.590164 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.590176    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wbp8\" (UniqueName: \"kubernetes.io/projected/542ca25a-e199-4923-bd3f-b39860cb678c-kube-api-access-5wbp8\") pod \"payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5\" (UID: \"542ca25a-e199-4923-bd3f-b39860cb678c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"
Oct 30 13:02:32.591478 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.591456    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542ca25a-e199-4923-bd3f-b39860cb678c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5\" (UID: \"542ca25a-e199-4923-bd3f-b39860cb678c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"
Oct 30 13:02:32.628498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.628464    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wbp8\" (UniqueName: \"kubernetes.io/projected/542ca25a-e199-4923-bd3f-b39860cb678c-kube-api-access-5wbp8\") pod \"payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5\" (UID: \"542ca25a-e199-4923-bd3f-b39860cb678c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"
Oct 30 13:02:32.692296 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.692201    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d630a8a0-5896-415e-8c51-34bcd9486545-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f\" (UID: \"d630a8a0-5896-415e-8c51-34bcd9486545\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"
Oct 30 13:02:32.692712 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.692687    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wchgn\" (UniqueName: \"kubernetes.io/projected/d630a8a0-5896-415e-8c51-34bcd9486545-kube-api-access-wchgn\") pod \"payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f\" (UID: \"d630a8a0-5896-415e-8c51-34bcd9486545\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"
Oct 30 13:02:32.753753 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.753718    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"
Oct 30 13:02:32.798015 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.796501    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wchgn\" (UniqueName: \"kubernetes.io/projected/d630a8a0-5896-415e-8c51-34bcd9486545-kube-api-access-wchgn\") pod \"payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f\" (UID: \"d630a8a0-5896-415e-8c51-34bcd9486545\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"
Oct 30 13:02:32.810115 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.810085    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d630a8a0-5896-415e-8c51-34bcd9486545-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f\" (UID: \"d630a8a0-5896-415e-8c51-34bcd9486545\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"
Oct 30 13:02:32.811898 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.811867    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d630a8a0-5896-415e-8c51-34bcd9486545-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f\" (UID: \"d630a8a0-5896-415e-8c51-34bcd9486545\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"
Oct 30 13:02:32.839534 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.838918    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wchgn\" (UniqueName: \"kubernetes.io/projected/d630a8a0-5896-415e-8c51-34bcd9486545-kube-api-access-wchgn\") pod \"payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f\" (UID: \"d630a8a0-5896-415e-8c51-34bcd9486545\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"
Oct 30 13:02:32.883676 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:32.883628    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"
Oct 30 13:02:33.145813 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.145782    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"]
Oct 30 13:02:33.147705 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:02:33.147675    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542ca25a_e199_4923_bd3f_b39860cb678c.slice/crio-ddc78ce4157441c2e2260dc8e04908a3521ee49ae0f8009a9e3986736b6b79a7 WatchSource:0}: Error finding container ddc78ce4157441c2e2260dc8e04908a3521ee49ae0f8009a9e3986736b6b79a7: Status 404 returned error can't find the container with id ddc78ce4157441c2e2260dc8e04908a3521ee49ae0f8009a9e3986736b6b79a7
Oct 30 13:02:33.217535 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.217509    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"]
Oct 30 13:02:33.225128 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.225103    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.249180 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.249155    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ba9a8c-openshift-default-dockercfg-k894v\""
Oct 30 13:02:33.254353 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.254324    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"]
Oct 30 13:02:33.298290 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.298215    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"]
Oct 30 13:02:33.392474 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.350707    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-workload-socket\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.392474 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.350845    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndpgl\" (UniqueName: \"kubernetes.io/projected/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-kube-api-access-ndpgl\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.392474 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.350883    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-token\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.392474 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.351014    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-envoy\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.392474 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.351068    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istiod-ca-cert\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.392474 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.351148    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-credential-socket\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.392474 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.351202    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-podinfo\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.392474 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.351236    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-data\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.392474 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.351298    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-workload-certs\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.454038 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.453996    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndpgl\" (UniqueName: \"kubernetes.io/projected/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-kube-api-access-ndpgl\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.454984 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.454958    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-token\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.456298 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.455549    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-envoy\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.456571 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.456555    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istiod-ca-cert\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.457380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.457236    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istiod-ca-cert\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.457804 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.457764    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-credential-socket\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.457898 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.457884    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-credential-socket\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.458034 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.458022    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-podinfo\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.458571 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.458555    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-data\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.459053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.458926    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-data\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.459197 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.459183    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-workload-certs\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.459674 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.459539    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-workload-certs\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.460122 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.460067    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-workload-socket\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.460235 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.460218    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-workload-socket\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.464096 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.464049    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-podinfo\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.503473 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.503434    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-envoy\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.507956 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.507925    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-token\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.516445 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.515526    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndpgl\" (UniqueName: \"kubernetes.io/projected/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-kube-api-access-ndpgl\") pod \"e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") " pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.572841 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.572768    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:33.905792 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:02:33.905730    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4600132d_6b1f_4f1e_bc28_76e86ee9b25f.slice/crio-763b3406cbf78fb6d57bb7dcd5bf668687d35fa6159f8490b5031c8ad11397b6 WatchSource:0}: Error finding container 763b3406cbf78fb6d57bb7dcd5bf668687d35fa6159f8490b5031c8ad11397b6: Status 404 returned error can't find the container with id 763b3406cbf78fb6d57bb7dcd5bf668687d35fa6159f8490b5031c8ad11397b6
Oct 30 13:02:33.906699 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.906648    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"]
Oct 30 13:02:33.909632 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.909591    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:33.909796 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.909675    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:33.909796 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:33.909716    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:34.080541 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:34.080514    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5" event={"ID":"542ca25a-e199-4923-bd3f-b39860cb678c","Type":"ContainerStarted","Data":"9ff4f9f1fde9f73840f6efe0112e367d03399df124d051bb9f9279a3afae6fc3"}
Oct 30 13:02:34.080771 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:34.080756    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5" event={"ID":"542ca25a-e199-4923-bd3f-b39860cb678c","Type":"ContainerStarted","Data":"ddc78ce4157441c2e2260dc8e04908a3521ee49ae0f8009a9e3986736b6b79a7"}
Oct 30 13:02:34.080930 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:34.080908    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"
Oct 30 13:02:34.082850 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:34.082824    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw" event={"ID":"4600132d-6b1f-4f1e-bc28-76e86ee9b25f","Type":"ContainerStarted","Data":"3f9214571706bed6ecb5f49e021aab15e8cc200c98667c5de21debf8fc6aec6f"}
Oct 30 13:02:34.083336 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:34.083317    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw" event={"ID":"4600132d-6b1f-4f1e-bc28-76e86ee9b25f","Type":"ContainerStarted","Data":"763b3406cbf78fb6d57bb7dcd5bf668687d35fa6159f8490b5031c8ad11397b6"}
Oct 30 13:02:34.085184 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:34.085161    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f" event={"ID":"d630a8a0-5896-415e-8c51-34bcd9486545","Type":"ContainerStarted","Data":"4a7fc0047b284c3217df98aa5bcad6d6dd3251c9c74fd44e10a2cf17719e2250"}
Oct 30 13:02:34.085284 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:34.085206    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f" event={"ID":"d630a8a0-5896-415e-8c51-34bcd9486545","Type":"ContainerStarted","Data":"87df5f6c7fadcff8cbf4b021a7f92810c1c89ae5433c31a16232ad39bbcae7d2"}
Oct 30 13:02:34.087643 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:34.087628    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"
Oct 30 13:02:34.121452 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:34.119617    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5" podStartSLOduration=2.119596038 podStartE2EDuration="2.119596038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:34.117543702 +0000 UTC m=+1542.641827935" watchObservedRunningTime="2026-10-30 13:02:34.119596038 +0000 UTC m=+1542.643880258"
Oct 30 13:02:34.164342 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:34.164238    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw" podStartSLOduration=1.164219777 podStartE2EDuration="1.164219777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:34.163176777 +0000 UTC m=+1542.687460997" watchObservedRunningTime="2026-10-30 13:02:34.164219777 +0000 UTC m=+1542.688503998"
Oct 30 13:02:34.206556 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:34.206497    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f" podStartSLOduration=2.206477595 podStartE2EDuration="2.206477595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:34.201309458 +0000 UTC m=+1542.725593678" watchObservedRunningTime="2026-10-30 13:02:34.206477595 +0000 UTC m=+1542.730761818"
Oct 30 13:02:34.573315 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:34.573283    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:35.579293 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:35.579266    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:36.110825 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:36.110789    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:36.111816 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:36.111791    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:02:37.355100 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.355079    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:37.434997 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.434962    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-workload-certs\") pod \"7543408a-97b7-4a86-bfaf-9834106505bf\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") "
Oct 30 13:02:37.434997 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.435001    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-credential-socket\") pod \"7543408a-97b7-4a86-bfaf-9834106505bf\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") "
Oct 30 13:02:37.435238 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.435128    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-istio-envoy\") pod \"7543408a-97b7-4a86-bfaf-9834106505bf\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") "
Oct 30 13:02:37.435323 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.435308    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7543408a-97b7-4a86-bfaf-9834106505bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7543408a-97b7-4a86-bfaf-9834106505bf-istio-podinfo\") pod \"7543408a-97b7-4a86-bfaf-9834106505bf\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") "
Oct 30 13:02:37.436501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.435389    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7543408a-97b7-4a86-bfaf-9834106505bf-kube-api-access-9hbxc\" (UniqueName: \"kubernetes.io/projected/7543408a-97b7-4a86-bfaf-9834106505bf-kube-api-access-9hbxc\") pod \"7543408a-97b7-4a86-bfaf-9834106505bf\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") "
Oct 30 13:02:37.436501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.435473    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-workload-socket\") pod \"7543408a-97b7-4a86-bfaf-9834106505bf\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") "
Oct 30 13:02:37.436501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.435534    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7543408a-97b7-4a86-bfaf-9834106505bf-istio-token\" (UniqueName: \"kubernetes.io/projected/7543408a-97b7-4a86-bfaf-9834106505bf-istio-token\") pod \"7543408a-97b7-4a86-bfaf-9834106505bf\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") "
Oct 30 13:02:37.436501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.435606    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7543408a-97b7-4a86-bfaf-9834106505bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7543408a-97b7-4a86-bfaf-9834106505bf-istiod-ca-cert\") pod \"7543408a-97b7-4a86-bfaf-9834106505bf\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") "
Oct 30 13:02:37.436501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.435648    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-istio-data\") pod \"7543408a-97b7-4a86-bfaf-9834106505bf\" (UID: \"7543408a-97b7-4a86-bfaf-9834106505bf\") "
Oct 30 13:02:37.436501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.435299    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-workload-certs" pod "7543408a-97b7-4a86-bfaf-9834106505bf" (UID: "7543408a-97b7-4a86-bfaf-9834106505bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:37.436501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.435382    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-credential-socket" pod "7543408a-97b7-4a86-bfaf-9834106505bf" (UID: "7543408a-97b7-4a86-bfaf-9834106505bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:37.436501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.436019    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-workload-socket" pod "7543408a-97b7-4a86-bfaf-9834106505bf" (UID: "7543408a-97b7-4a86-bfaf-9834106505bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:37.436501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.436099    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-istio-data" pod "7543408a-97b7-4a86-bfaf-9834106505bf" (UID: "7543408a-97b7-4a86-bfaf-9834106505bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:37.436501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.436254    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:37.436501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.436273    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:37.436501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.436281    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:37.436501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.436289    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:37.436501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.436445    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7543408a-97b7-4a86-bfaf-9834106505bf-istiod-ca-cert" pod "7543408a-97b7-4a86-bfaf-9834106505bf" (UID: "7543408a-97b7-4a86-bfaf-9834106505bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:37.438275 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.438244    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-istio-envoy" pod "7543408a-97b7-4a86-bfaf-9834106505bf" (UID: "7543408a-97b7-4a86-bfaf-9834106505bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:37.442164 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.442137    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7543408a-97b7-4a86-bfaf-9834106505bf-istio-token" pod "7543408a-97b7-4a86-bfaf-9834106505bf" (UID: "7543408a-97b7-4a86-bfaf-9834106505bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:37.442578 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.442550    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7543408a-97b7-4a86-bfaf-9834106505bf-istio-podinfo" pod "7543408a-97b7-4a86-bfaf-9834106505bf" (UID: "7543408a-97b7-4a86-bfaf-9834106505bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:02:37.442678 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.442581    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7543408a-97b7-4a86-bfaf-9834106505bf-kube-api-access-9hbxc" pod "7543408a-97b7-4a86-bfaf-9834106505bf" (UID: "7543408a-97b7-4a86-bfaf-9834106505bf"). InnerVolumeSpecName "kube-api-access-9hbxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:37.537387 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.537353    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7543408a-97b7-4a86-bfaf-9834106505bf-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:37.537387 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.537383    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7543408a-97b7-4a86-bfaf-9834106505bf-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:37.537387 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.537419    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7543408a-97b7-4a86-bfaf-9834106505bf-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:37.537387 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.537427    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7543408a-97b7-4a86-bfaf-9834106505bf-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:37.537681 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:37.537438    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hbxc\" (UniqueName: \"kubernetes.io/projected/7543408a-97b7-4a86-bfaf-9834106505bf-kube-api-access-9hbxc\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:02:38.128003 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:38.127966    5837 generic.go:356] "Generic (PLEG): container finished" podID="7543408a-97b7-4a86-bfaf-9834106505bf" containerID="c6fddaa6d412f069419f505ae7b6a707955cdef8a8f03aec8e507917ee6b7f6a" exitCode=0
Oct 30 13:02:38.128207 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:38.128019    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"
Oct 30 13:02:38.128207 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:38.128048    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8" event={"ID":"7543408a-97b7-4a86-bfaf-9834106505bf","Type":"ContainerDied","Data":"c6fddaa6d412f069419f505ae7b6a707955cdef8a8f03aec8e507917ee6b7f6a"}
Oct 30 13:02:38.128207 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:38.128110    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8" event={"ID":"7543408a-97b7-4a86-bfaf-9834106505bf","Type":"ContainerDied","Data":"2bb1044c225434063b9e121aec17daaa4f2d4124cf6b522326f35be5dcbb476c"}
Oct 30 13:02:38.128207 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:38.128131    5837 scope.go:122] "RemoveContainer" containerID="c6fddaa6d412f069419f505ae7b6a707955cdef8a8f03aec8e507917ee6b7f6a"
Oct 30 13:02:38.139279 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:38.139260    5837 scope.go:122] "RemoveContainer" containerID="c6fddaa6d412f069419f505ae7b6a707955cdef8a8f03aec8e507917ee6b7f6a"
Oct 30 13:02:38.139647 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:02:38.139619    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6fddaa6d412f069419f505ae7b6a707955cdef8a8f03aec8e507917ee6b7f6a\": container with ID starting with c6fddaa6d412f069419f505ae7b6a707955cdef8a8f03aec8e507917ee6b7f6a not found: ID does not exist" containerID="c6fddaa6d412f069419f505ae7b6a707955cdef8a8f03aec8e507917ee6b7f6a"
Oct 30 13:02:38.139762 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:38.139650    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6fddaa6d412f069419f505ae7b6a707955cdef8a8f03aec8e507917ee6b7f6a"} err="failed to get container status \"c6fddaa6d412f069419f505ae7b6a707955cdef8a8f03aec8e507917ee6b7f6a\": rpc error: code = NotFound desc = could not find container \"c6fddaa6d412f069419f505ae7b6a707955cdef8a8f03aec8e507917ee6b7f6a\": container with ID starting with c6fddaa6d412f069419f505ae7b6a707955cdef8a8f03aec8e507917ee6b7f6a not found: ID does not exist"
Oct 30 13:02:38.191380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:38.191355    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"]
Oct 30 13:02:38.203426 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:38.203386    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-zb9k8"]
Oct 30 13:02:38.985890 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:38.985854    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"]
Oct 30 13:02:38.986566 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:38.986552    5837 cpu_manager.go:414] "containerMap: removing container" podUID="7543408a-97b7-4a86-bfaf-9834106505bf" containerName="istio-proxy"
Oct 30 13:02:38.986566 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:38.986566    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7543408a-97b7-4a86-bfaf-9834106505bf" containerName="istio-proxy"
Oct 30 13:02:38.986705 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:38.986696    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="7543408a-97b7-4a86-bfaf-9834106505bf" containerName="istio-proxy"
Oct 30 13:02:38.990447 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:38.990388    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"
Oct 30 13:02:39.008956 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.008925    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"]
Oct 30 13:02:39.015303 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.015260    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ba9a8c\""
Oct 30 13:02:39.015463 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.015425    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ba9a8c-dockercfg-qksnc\""
Oct 30 13:02:39.289477 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.281601    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"]
Oct 30 13:02:39.294136 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.294110    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"
Oct 30 13:02:39.301334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.298448    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c768w\" (UniqueName: \"kubernetes.io/projected/158886bc-5fe6-4182-b481-389fc298abd8-kube-api-access-c768w\") pod \"payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r\" (UID: \"158886bc-5fe6-4182-b481-389fc298abd8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"
Oct 30 13:02:39.301334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.298578    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158886bc-5fe6-4182-b481-389fc298abd8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r\" (UID: \"158886bc-5fe6-4182-b481-389fc298abd8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"
Oct 30 13:02:39.301334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.298668    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79da32ac-98d8-481a-9713-7e784f1c2ce7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt\" (UID: \"79da32ac-98d8-481a-9713-7e784f1c2ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"
Oct 30 13:02:39.301334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.298696    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thflw\" (UniqueName: \"kubernetes.io/projected/79da32ac-98d8-481a-9713-7e784f1c2ce7-kube-api-access-thflw\") pod \"payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt\" (UID: \"79da32ac-98d8-481a-9713-7e784f1c2ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"
Oct 30 13:02:39.301334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.299460    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"]
Oct 30 13:02:39.400624 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.400591    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c768w\" (UniqueName: \"kubernetes.io/projected/158886bc-5fe6-4182-b481-389fc298abd8-kube-api-access-c768w\") pod \"payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r\" (UID: \"158886bc-5fe6-4182-b481-389fc298abd8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"
Oct 30 13:02:39.400827 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.400741    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158886bc-5fe6-4182-b481-389fc298abd8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r\" (UID: \"158886bc-5fe6-4182-b481-389fc298abd8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"
Oct 30 13:02:39.400900 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.400856    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79da32ac-98d8-481a-9713-7e784f1c2ce7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt\" (UID: \"79da32ac-98d8-481a-9713-7e784f1c2ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"
Oct 30 13:02:39.400900 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.400882    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thflw\" (UniqueName: \"kubernetes.io/projected/79da32ac-98d8-481a-9713-7e784f1c2ce7-kube-api-access-thflw\") pod \"payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt\" (UID: \"79da32ac-98d8-481a-9713-7e784f1c2ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"
Oct 30 13:02:39.401460 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.401439    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158886bc-5fe6-4182-b481-389fc298abd8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r\" (UID: \"158886bc-5fe6-4182-b481-389fc298abd8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"
Oct 30 13:02:39.401536 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.401492    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79da32ac-98d8-481a-9713-7e784f1c2ce7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt\" (UID: \"79da32ac-98d8-481a-9713-7e784f1c2ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"
Oct 30 13:02:39.419162 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.419136    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thflw\" (UniqueName: \"kubernetes.io/projected/79da32ac-98d8-481a-9713-7e784f1c2ce7-kube-api-access-thflw\") pod \"payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt\" (UID: \"79da32ac-98d8-481a-9713-7e784f1c2ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"
Oct 30 13:02:39.419295 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.419188    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c768w\" (UniqueName: \"kubernetes.io/projected/158886bc-5fe6-4182-b481-389fc298abd8-kube-api-access-c768w\") pod \"payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r\" (UID: \"158886bc-5fe6-4182-b481-389fc298abd8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"
Oct 30 13:02:39.611414 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.611308    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"
Oct 30 13:02:39.635014 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.634970    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7543408a-97b7-4a86-bfaf-9834106505bf" path="/var/lib/kubelet/pods/7543408a-97b7-4a86-bfaf-9834106505bf/volumes"
Oct 30 13:02:39.646649 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.646575    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"
Oct 30 13:02:39.975171 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:39.974802    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"]
Oct 30 13:02:39.975439 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:02:39.975222    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79da32ac_98d8_481a_9713_7e784f1c2ce7.slice/crio-7fe431ad68ed7f16a2e4c21ce05e5c6c8e08e0ee30cf00508cac243d76b967ae WatchSource:0}: Error finding container 7fe431ad68ed7f16a2e4c21ce05e5c6c8e08e0ee30cf00508cac243d76b967ae: Status 404 returned error can't find the container with id 7fe431ad68ed7f16a2e4c21ce05e5c6c8e08e0ee30cf00508cac243d76b967ae
Oct 30 13:02:40.003363 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:40.003338    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"]
Oct 30 13:02:40.003981 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:02:40.003958    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158886bc_5fe6_4182_b481_389fc298abd8.slice/crio-5386381bdb420b828942935527a26e5f80f07e66c915a8493100cdec0bd09385 WatchSource:0}: Error finding container 5386381bdb420b828942935527a26e5f80f07e66c915a8493100cdec0bd09385: Status 404 returned error can't find the container with id 5386381bdb420b828942935527a26e5f80f07e66c915a8493100cdec0bd09385
Oct 30 13:02:40.441717 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:40.441681    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt" event={"ID":"79da32ac-98d8-481a-9713-7e784f1c2ce7","Type":"ContainerStarted","Data":"1af96a80cf6cc27d3925c4d3d79f86f21e6c32a1c83328427519765ccd64bb5a"}
Oct 30 13:02:40.441902 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:40.441742    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt" event={"ID":"79da32ac-98d8-481a-9713-7e784f1c2ce7","Type":"ContainerStarted","Data":"7fe431ad68ed7f16a2e4c21ce05e5c6c8e08e0ee30cf00508cac243d76b967ae"}
Oct 30 13:02:40.442007 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:40.441989    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"
Oct 30 13:02:40.445561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:40.445515    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r" event={"ID":"158886bc-5fe6-4182-b481-389fc298abd8","Type":"ContainerStarted","Data":"6b272fa59925e8567a19005dfc0dabaf9d388f24e5a36f5f31041dc0e3ae4325"}
Oct 30 13:02:40.445751 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:40.445588    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r" event={"ID":"158886bc-5fe6-4182-b481-389fc298abd8","Type":"ContainerStarted","Data":"5386381bdb420b828942935527a26e5f80f07e66c915a8493100cdec0bd09385"}
Oct 30 13:02:40.446585 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:40.446446    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"
Oct 30 13:02:40.482705 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:40.482657    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt" podStartSLOduration=2.482640374 podStartE2EDuration="2.482640374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:40.481539741 +0000 UTC m=+1549.005823960" watchObservedRunningTime="2026-10-30 13:02:40.482640374 +0000 UTC m=+1549.006924593"
Oct 30 13:02:40.523819 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:40.523753    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r" podStartSLOduration=1.523738703 podStartE2EDuration="1.523738703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:40.519013823 +0000 UTC m=+1549.043298050" watchObservedRunningTime="2026-10-30 13:02:40.523738703 +0000 UTC m=+1549.048022923"
Oct 30 13:02:45.095128 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:45.095099    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"
Oct 30 13:02:45.095871 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:02:45.095843    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:02:46.113205 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:46.113177    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"
Oct 30 13:02:51.497456 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:51.491990    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"
Oct 30 13:02:52.027632 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:52.027598    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"]
Oct 30 13:02:52.039024 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:52.038997    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b14f5479-ncjks"]
Oct 30 13:02:52.521115 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:52.521083    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"
Oct 30 13:02:53.606848 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:53.606817    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2983f91d-27e6-4289-80c6-0a5bb4162128" path="/var/lib/kubelet/pods/2983f91d-27e6-4289-80c6-0a5bb4162128/volumes"
Oct 30 13:02:56.935133 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:56.935095    5837 scope.go:122] "RemoveContainer" containerID="9f688b4495f602fbcc2dfbcaa21c1ae2fd71ba9fa08aafd74de7377a06142d8a"
Oct 30 13:02:56.979990 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:02:56.979959    5837 scope.go:122] "RemoveContainer" containerID="ab809a0f3fee82a625cc2e492d17b2d2859d838fbe21665f3c1192b171d34240"
Oct 30 13:03:07.624664 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.624633    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"]
Oct 30 13:03:07.631509 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.631482    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"
Oct 30 13:03:07.663730 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.663707    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"]
Oct 30 13:03:07.673445 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.670193    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:03:07.673445 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.670209    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:03:07.673445 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.670258    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:03:07.673445 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.670309    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:03:07.690490 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.690461    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc\" (UID: \"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"
Oct 30 13:03:07.690596 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.690503    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc\" (UID: \"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"
Oct 30 13:03:07.690596 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.690539    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ght6\" (UniqueName: \"kubernetes.io/projected/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-kube-api-access-5ght6\") pod \"maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc\" (UID: \"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"
Oct 30 13:03:07.791824 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.791793    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc\" (UID: \"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"
Oct 30 13:03:07.791984 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.791844    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc\" (UID: \"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"
Oct 30 13:03:07.791984 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.791895    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ght6\" (UniqueName: \"kubernetes.io/projected/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-kube-api-access-5ght6\") pod \"maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc\" (UID: \"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"
Oct 30 13:03:07.792638 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.792607    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc\" (UID: \"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"
Oct 30 13:03:07.794362 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.794341    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc\" (UID: \"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"
Oct 30 13:03:07.811516 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.811492    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ght6\" (UniqueName: \"kubernetes.io/projected/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-kube-api-access-5ght6\") pod \"maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc\" (UID: \"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"
Oct 30 13:03:07.946387 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:07.946323    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"
Oct 30 13:03:08.246873 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:03:08.246833    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c8cedf_2ff2_4f5b_a00d_36c4b7e94f08.slice/crio-cac6052bccfeb1444acf178de8cc01beb92095f460f006a4b42667741900a0a4 WatchSource:0}: Error finding container cac6052bccfeb1444acf178de8cc01beb92095f460f006a4b42667741900a0a4: Status 404 returned error can't find the container with id cac6052bccfeb1444acf178de8cc01beb92095f460f006a4b42667741900a0a4
Oct 30 13:03:08.248001 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:08.247980    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"]
Oct 30 13:03:08.836155 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:08.836083    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc" event={"ID":"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08","Type":"ContainerStarted","Data":"6369ed25c82ccd8c5ed6b22f4e73240fc38739b0121315da8a00750c1df93778"}
Oct 30 13:03:08.836155 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:08.836135    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc" event={"ID":"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08","Type":"ContainerStarted","Data":"cac6052bccfeb1444acf178de8cc01beb92095f460f006a4b42667741900a0a4"}
Oct 30 13:03:08.881525 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:08.881477    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc" podStartSLOduration=1.88146433 podStartE2EDuration="1.88146433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:08.877257476 +0000 UTC m=+1577.401541695" watchObservedRunningTime="2026-10-30 13:03:08.88146433 +0000 UTC m=+1577.405748549"
Oct 30 13:03:09.845307 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:09.845276    5837 generic.go:356] "Generic (PLEG): container finished" podID="d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08" containerID="6369ed25c82ccd8c5ed6b22f4e73240fc38739b0121315da8a00750c1df93778" exitCode=0
Oct 30 13:03:09.845708 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:09.845332    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc" event={"ID":"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08","Type":"ContainerDied","Data":"6369ed25c82ccd8c5ed6b22f4e73240fc38739b0121315da8a00750c1df93778"}
Oct 30 13:03:11.025066 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:11.025040    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"
Oct 30 13:03:11.143770 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:11.143691    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-kube-api-access-5ght6\" (UniqueName: \"kubernetes.io/projected/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-kube-api-access-5ght6\") pod \"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08\" (UID: \"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08\") "
Oct 30 13:03:11.143770 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:11.143755    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-maas-api-xks-ca\") pod \"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08\" (UID: \"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08\") "
Oct 30 13:03:11.143979 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:11.143871    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-maas-api-openshift-ca\") pod \"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08\" (UID: \"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08\") "
Oct 30 13:03:11.144276 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:11.144251    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-maas-api-openshift-ca" pod "d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08" (UID: "d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:11.144652 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:11.144630    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:11.146022 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:11.145998    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-kube-api-access-5ght6" pod "d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08" (UID: "d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08"). InnerVolumeSpecName "kube-api-access-5ght6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:11.146022 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:11.146005    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-maas-api-xks-ca" pod "d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08" (UID: "d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:03:11.246299 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:11.246267    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ght6\" (UniqueName: \"kubernetes.io/projected/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-kube-api-access-5ght6\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:11.246299 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:11.246296    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:11.860680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:11.860653    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"
Oct 30 13:03:11.860855 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:11.860689    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc" event={"ID":"d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08","Type":"ContainerDied","Data":"cac6052bccfeb1444acf178de8cc01beb92095f460f006a4b42667741900a0a4"}
Oct 30 13:03:11.860855 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:11.860744    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cac6052bccfeb1444acf178de8cc01beb92095f460f006a4b42667741900a0a4"
Oct 30 13:03:15.097938 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:03:15.097901    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:03:15.104544 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:15.104517    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"]
Oct 30 13:03:15.113656 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:15.113629    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45b2b131-4m4vs"]
Oct 30 13:03:15.607009 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:15.606980    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5255ee1e-35fa-45a2-bd3a-b645167e6b44" path="/var/lib/kubelet/pods/5255ee1e-35fa-45a2-bd3a-b645167e6b44/volumes"
Oct 30 13:03:18.296153 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:18.296106    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"]
Oct 30 13:03:20.284458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:20.280504    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:21.291193 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:21.291147    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:21.292349 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:21.292324    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 13:03:21.911422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:21.883104    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 13:03:22.138570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:22.138525    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:22.524120 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:22.523621    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:22.524120 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:22.523702    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:22.884558 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:22.881289    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:23.200696 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:23.200598    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:23.202074 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:23.202049    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 13:03:23.297378 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:23.297334    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm" podUID="fb9bc72c-02a4-4edb-8534-f3b5d62a3882" containerName="payload-processing" containerID="cri-o://aa3cf4fb19f4a187d55cf319ca801c98c854f13f83108d6decd92803efd718c2" gracePeriod=25
Oct 30 13:03:23.410436 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:23.363021    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:23.915433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:23.908499    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:03:23.915433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:23.909030    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 13:03:24.211478 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:24.208538    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.365576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:24.365517    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.392435 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:24.377653    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:03:24.584453 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:24.563792    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.947472 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:24.947147    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.968782 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:24.948509    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq" podUID="b175788a-b7a3-4aab-a17f-1362ac6b83b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:25.045431 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:24.988250    5837 generic.go:356] "Generic (PLEG): container finished" podID="fb9bc72c-02a4-4edb-8534-f3b5d62a3882" containerID="aa3cf4fb19f4a187d55cf319ca801c98c854f13f83108d6decd92803efd718c2" exitCode=0
Oct 30 13:03:25.045782 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:25.008881    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm" event={"ID":"fb9bc72c-02a4-4edb-8534-f3b5d62a3882","Type":"ContainerDied","Data":"aa3cf4fb19f4a187d55cf319ca801c98c854f13f83108d6decd92803efd718c2"}
Oct 30 13:03:25.247421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:25.239957    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:25.367419 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:25.366514    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:25.573474 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:25.572689    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:25.573474 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:25.572782    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:25.790869 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:25.744527    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:25.790869 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:25.744623    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" podUID="faf87439-5044-43cd-9b1a-200188299f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:25.917434 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:25.916890    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:25.917434 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:25.916957    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:26.119057 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:26.079461    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:26.255536 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:26.246074    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:26.272433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:26.268536    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:26.272433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:26.268598    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" podUID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:26.410608 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:26.367078    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:03:26.760779 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:26.760746    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"
Oct 30 13:03:26.881422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:26.879040    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9bc72c-02a4-4edb-8534-f3b5d62a3882-kube-api-access-lkdqr\" (UniqueName: \"kubernetes.io/projected/fb9bc72c-02a4-4edb-8534-f3b5d62a3882-kube-api-access-lkdqr\") pod \"fb9bc72c-02a4-4edb-8534-f3b5d62a3882\" (UID: \"fb9bc72c-02a4-4edb-8534-f3b5d62a3882\") "
Oct 30 13:03:26.881422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:26.879281    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb9bc72c-02a4-4edb-8534-f3b5d62a3882-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9bc72c-02a4-4edb-8534-f3b5d62a3882-plugins-config-volume\") pod \"fb9bc72c-02a4-4edb-8534-f3b5d62a3882\" (UID: \"fb9bc72c-02a4-4edb-8534-f3b5d62a3882\") "
Oct 30 13:03:26.892473 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:26.892372    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9bc72c-02a4-4edb-8534-f3b5d62a3882-kube-api-access-lkdqr" pod "fb9bc72c-02a4-4edb-8534-f3b5d62a3882" (UID: "fb9bc72c-02a4-4edb-8534-f3b5d62a3882"). InnerVolumeSpecName "kube-api-access-lkdqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:26.952432 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:26.948903    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9bc72c-02a4-4edb-8534-f3b5d62a3882-plugins-config-volume" pod "fb9bc72c-02a4-4edb-8534-f3b5d62a3882" (UID: "fb9bc72c-02a4-4edb-8534-f3b5d62a3882"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:26.991448 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:26.989105    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkdqr\" (UniqueName: \"kubernetes.io/projected/fb9bc72c-02a4-4edb-8534-f3b5d62a3882-kube-api-access-lkdqr\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:26.991448 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:26.989144    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9bc72c-02a4-4edb-8534-f3b5d62a3882-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:27.123421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:27.106831    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:03:27.277547 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:27.277505    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:27.384221 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:27.384182    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:27.617268 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:27.616804    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-snzms" podUID="b35e847c-8794-4355-8d45-f474df36feee" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:28.136428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:28.128431    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:03:28.314431 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:28.306532    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:28.460480 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:28.444667    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:28.566821 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:28.566777    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm" event={"ID":"fb9bc72c-02a4-4edb-8534-f3b5d62a3882","Type":"ContainerDied","Data":"c90da873000b35b212976306a5b7332b410a0c25cea10b23e34aa28388c268b2"}
Oct 30 13:03:28.570419 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:28.567148    5837 scope.go:122] "RemoveContainer" containerID="aa3cf4fb19f4a187d55cf319ca801c98c854f13f83108d6decd92803efd718c2"
Oct 30 13:03:28.581798 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:28.570852    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"
Oct 30 13:03:28.887827 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:28.887748    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"]
Oct 30 13:03:28.989506 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:28.942378    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-87f1b5-79ccf88478-njqdm"]
Oct 30 13:03:29.173481 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:29.172563    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:29.343422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:29.334528    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 13:03:29.460673 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:29.460551    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:03:29.544584 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:29.544544    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": context deadline exceeded"
Oct 30 13:03:29.795458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:29.777350    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb9bc72c-02a4-4edb-8534-f3b5d62a3882" path="/var/lib/kubelet/pods/fb9bc72c-02a4-4edb-8534-f3b5d62a3882/volumes"
Oct 30 13:03:29.838829 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:29.835586    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 13:03:30.334914 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:30.334885    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:03:30.350176 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:30.349533    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 13:03:34.306060 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.306015    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"]
Oct 30 13:03:34.307159 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.307134    5837 cpu_manager.go:414] "containerMap: removing container" podUID="fb9bc72c-02a4-4edb-8534-f3b5d62a3882" containerName="payload-processing"
Oct 30 13:03:34.307348 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.307329    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9bc72c-02a4-4edb-8534-f3b5d62a3882" containerName="payload-processing"
Oct 30 13:03:34.307518 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.307505    5837 cpu_manager.go:414] "containerMap: removing container" podUID="d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08" containerName="revoke-keys"
Oct 30 13:03:34.307615 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.307603    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08" containerName="revoke-keys"
Oct 30 13:03:34.307934 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.307919    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9bc72c-02a4-4edb-8534-f3b5d62a3882" containerName="payload-processing"
Oct 30 13:03:34.308038 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.308028    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08" containerName="revoke-keys"
Oct 30 13:03:34.328369 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.327295    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.330379 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.330355    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"]
Oct 30 13:03:34.354878 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.354847    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-1f91c1-openshift-default-dockercfg-j52fr\""
Oct 30 13:03:34.439104 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.424803    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htnsz\" (UniqueName: \"kubernetes.io/projected/f163e2e9-ace0-44b6-962c-19b61fe31c95-kube-api-access-htnsz\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.439104 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.424896    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f163e2e9-ace0-44b6-962c-19b61fe31c95-istiod-ca-cert\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.439104 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.424990    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-data\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.439104 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.425033    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-envoy\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.439104 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.425082    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-credential-socket\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.439104 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.425180    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-token\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.439104 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.425242    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-workload-socket\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.439104 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.425272    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-workload-certs\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.439104 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.425462    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-podinfo\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.573470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.529491    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htnsz\" (UniqueName: \"kubernetes.io/projected/f163e2e9-ace0-44b6-962c-19b61fe31c95-kube-api-access-htnsz\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.573470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.529560    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f163e2e9-ace0-44b6-962c-19b61fe31c95-istiod-ca-cert\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.573470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.529635    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-data\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.573470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.529672    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-envoy\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.573470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.529717    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-credential-socket\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.573470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.529752    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-token\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.573470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.529784    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-workload-socket\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.573470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.529814    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-workload-certs\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.573470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.529962    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-podinfo\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.609110 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.609062    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-credential-socket\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.629459 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.609608    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-workload-socket\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.629741 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.611307    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-workload-certs\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.629871 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.613006    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f163e2e9-ace0-44b6-962c-19b61fe31c95-istiod-ca-cert\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.629963 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.613275    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-data\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.658819 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.658751    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-envoy\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.668533 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.668495    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-token\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.669294 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.669265    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-podinfo\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.692571 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.673482    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htnsz\" (UniqueName: \"kubernetes.io/projected/f163e2e9-ace0-44b6-962c-19b61fe31c95-kube-api-access-htnsz\") pod \"e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") " pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:34.694112 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:34.693860    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:35.077977 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:35.077948    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"]
Oct 30 13:03:35.078574 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:03:35.078541    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf163e2e9_ace0_44b6_962c_19b61fe31c95.slice/crio-ce708932df8c1d7501e2313ffad59c488387ff06f7379341b10decb9aefbeb01 WatchSource:0}: Error finding container ce708932df8c1d7501e2313ffad59c488387ff06f7379341b10decb9aefbeb01: Status 404 returned error can't find the container with id ce708932df8c1d7501e2313ffad59c488387ff06f7379341b10decb9aefbeb01
Oct 30 13:03:35.081252 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:35.081220    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:35.081344 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:35.081289    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:35.081344 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:35.081321    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:35.754686 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:35.754612    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" event={"ID":"f163e2e9-ace0-44b6-962c-19b61fe31c95","Type":"ContainerStarted","Data":"3f11e4b460fb43ff4546577e286bbb4010d22453f87ba60ea169e58a6e0f7f48"}
Oct 30 13:03:35.755210 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:35.754674    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" event={"ID":"f163e2e9-ace0-44b6-962c-19b61fe31c95","Type":"ContainerStarted","Data":"ce708932df8c1d7501e2313ffad59c488387ff06f7379341b10decb9aefbeb01"}
Oct 30 13:03:35.803626 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:35.803577    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" podStartSLOduration=1.803561367 podStartE2EDuration="1.803561367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:35.798009901 +0000 UTC m=+1604.322294122" watchObservedRunningTime="2026-10-30 13:03:35.803561367 +0000 UTC m=+1604.327845583"
Oct 30 13:03:36.694465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:36.694425    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:36.700408 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:36.700366    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:36.763143 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:36.763115    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:36.764247 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:36.764228    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:03:37.152607 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:37.152562    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:37.152801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:37.152659    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw" podUID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:38.802197 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:38.786814    5837 generic.go:356] "Generic (PLEG): container finished" podID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerID="ff6b665d8257d32f953ec76d48bf420e6fff2572de455306840fc3cb4bed198a" exitCode=0
Oct 30 13:03:38.802197 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:38.787893    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" event={"ID":"9e2ab740-6bfa-438b-a878-e8a852a9b1a1","Type":"ContainerDied","Data":"ff6b665d8257d32f953ec76d48bf420e6fff2572de455306840fc3cb4bed198a"}
Oct 30 13:03:38.802197 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:38.788346    5837 scope.go:122] "RemoveContainer" containerID="ff6b665d8257d32f953ec76d48bf420e6fff2572de455306840fc3cb4bed198a"
Oct 30 13:03:38.802197 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:38.790491    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:38.802197 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:38.790574    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:38.802197 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:38.790621    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:39.796445 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:39.796406    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" event={"ID":"9e2ab740-6bfa-438b-a878-e8a852a9b1a1","Type":"ContainerStarted","Data":"23bffeb0690db510e246651f4b333fec203c44b99fba02d4f29b7bf6feab5d1b"}
Oct 30 13:03:39.798316 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:39.798288    5837 generic.go:356] "Generic (PLEG): container finished" podID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerID="3f9214571706bed6ecb5f49e021aab15e8cc200c98667c5de21debf8fc6aec6f" exitCode=0
Oct 30 13:03:39.798495 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:39.798354    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw" event={"ID":"4600132d-6b1f-4f1e-bc28-76e86ee9b25f","Type":"ContainerDied","Data":"3f9214571706bed6ecb5f49e021aab15e8cc200c98667c5de21debf8fc6aec6f"}
Oct 30 13:03:39.798913 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:39.798896    5837 scope.go:122] "RemoveContainer" containerID="3f9214571706bed6ecb5f49e021aab15e8cc200c98667c5de21debf8fc6aec6f"
Oct 30 13:03:39.801199 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:39.801168    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:39.801307 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:39.801239    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:39.801307 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:39.801277    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:40.319679 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.319646    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"]
Oct 30 13:03:40.323948 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.323925    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"
Oct 30 13:03:40.340125 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.340105    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-1f91c1\""
Oct 30 13:03:40.340253 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.340168    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-1f91c1-dockercfg-pr9zd\""
Oct 30 13:03:40.342312 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.342291    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"]
Oct 30 13:03:40.358629 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.358595    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188cba42-8f91-4d73-a8be-858791be060b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp\" (UID: \"188cba42-8f91-4d73-a8be-858791be060b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"
Oct 30 13:03:40.358752 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.358680    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbntz\" (UniqueName: \"kubernetes.io/projected/188cba42-8f91-4d73-a8be-858791be060b-kube-api-access-bbntz\") pod \"payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp\" (UID: \"188cba42-8f91-4d73-a8be-858791be060b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"
Oct 30 13:03:40.361421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.361382    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:03:40.361536 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.361504    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:03:40.361611 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.361544    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 13:03:40.383310 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.382870    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"]
Oct 30 13:03:40.473419 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.469611    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188cba42-8f91-4d73-a8be-858791be060b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp\" (UID: \"188cba42-8f91-4d73-a8be-858791be060b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"
Oct 30 13:03:40.473419 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.469742    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbntz\" (UniqueName: \"kubernetes.io/projected/188cba42-8f91-4d73-a8be-858791be060b-kube-api-access-bbntz\") pod \"payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp\" (UID: \"188cba42-8f91-4d73-a8be-858791be060b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"
Oct 30 13:03:40.473419 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.471334    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188cba42-8f91-4d73-a8be-858791be060b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp\" (UID: \"188cba42-8f91-4d73-a8be-858791be060b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"
Oct 30 13:03:40.494466 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.493755    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"
Oct 30 13:03:40.559420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.552454    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbntz\" (UniqueName: \"kubernetes.io/projected/188cba42-8f91-4d73-a8be-858791be060b-kube-api-access-bbntz\") pod \"payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp\" (UID: \"188cba42-8f91-4d73-a8be-858791be060b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"
Oct 30 13:03:40.601416 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.592378    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd578\" (UniqueName: \"kubernetes.io/projected/2a03f1bb-dfc9-4e35-9723-8ec01432a544-kube-api-access-wd578\") pod \"payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4\" (UID: \"2a03f1bb-dfc9-4e35-9723-8ec01432a544\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"
Oct 30 13:03:40.601416 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.592502    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a03f1bb-dfc9-4e35-9723-8ec01432a544-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4\" (UID: \"2a03f1bb-dfc9-4e35-9723-8ec01432a544\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"
Oct 30 13:03:40.672927 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.672685    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"]
Oct 30 13:03:40.674409 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.674140    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"
Oct 30 13:03:40.756733 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.756698    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a03f1bb-dfc9-4e35-9723-8ec01432a544-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4\" (UID: \"2a03f1bb-dfc9-4e35-9723-8ec01432a544\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"
Oct 30 13:03:40.757209 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.757166    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd578\" (UniqueName: \"kubernetes.io/projected/2a03f1bb-dfc9-4e35-9723-8ec01432a544-kube-api-access-wd578\") pod \"payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4\" (UID: \"2a03f1bb-dfc9-4e35-9723-8ec01432a544\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"
Oct 30 13:03:40.758772 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.758738    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a03f1bb-dfc9-4e35-9723-8ec01432a544-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4\" (UID: \"2a03f1bb-dfc9-4e35-9723-8ec01432a544\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"
Oct 30 13:03:40.779972 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.779929    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd578\" (UniqueName: \"kubernetes.io/projected/2a03f1bb-dfc9-4e35-9723-8ec01432a544-kube-api-access-wd578\") pod \"payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4\" (UID: \"2a03f1bb-dfc9-4e35-9723-8ec01432a544\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"
Oct 30 13:03:40.859262 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:40.858065    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw" event={"ID":"4600132d-6b1f-4f1e-bc28-76e86ee9b25f","Type":"ContainerStarted","Data":"1f9605a12a898d1943bd473ac3272ae1ca6ccb723afdacee30e6e0a0b36de7a9"}
Oct 30 13:03:41.059458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:41.054281    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"
Oct 30 13:03:41.143857 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:41.143813    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"]
Oct 30 13:03:41.361981 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:41.361887    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:03:41.361981 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:41.361941    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 13:03:41.611062 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:41.590994    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:03:41.611062 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:41.591127    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:03:41.611062 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:41.591168    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw" podUID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 13:03:41.611062 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:41.606442    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"]
Oct 30 13:03:42.076051 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:42.075998    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4" event={"ID":"2a03f1bb-dfc9-4e35-9723-8ec01432a544","Type":"ContainerStarted","Data":"dda61d42b5129e5749b7fee5c2fd235d3e089e6c6cf5850a2acfb1d611e8caa2"}
Oct 30 13:03:42.081209 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:42.081161    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp" event={"ID":"188cba42-8f91-4d73-a8be-858791be060b","Type":"ContainerStarted","Data":"c6a0aa91b41059ea8c80ff1efba24d94659cb828d37b1ceb1633734ed1f5d5e3"}
Oct 30 13:03:42.081635 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:42.081614    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp" event={"ID":"188cba42-8f91-4d73-a8be-858791be060b","Type":"ContainerStarted","Data":"b042e0e1bbcd5f37bf15fefb9149104bbeadc33d31cbf3ffc838a5edb2696c38"}
Oct 30 13:03:42.082121 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:42.082103    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"
Oct 30 13:03:42.156934 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:42.154829    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp" podStartSLOduration=2.154805455 podStartE2EDuration="2.154805455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:42.149063356 +0000 UTC m=+1610.673347579" watchObservedRunningTime="2026-10-30 13:03:42.154805455 +0000 UTC m=+1610.679089674"
Oct 30 13:03:42.362104 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:42.362069    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:03:42.362518 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:42.362121    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 13:03:42.402471 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:42.401113    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"]
Oct 30 13:03:42.525419 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:42.506913    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87f1b5-fp9kc"]
Oct 30 13:03:42.574155 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:42.574122    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:03:42.574442 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:42.574415    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw" podUID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 13:03:43.121488 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:43.121449    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4" event={"ID":"2a03f1bb-dfc9-4e35-9723-8ec01432a544","Type":"ContainerStarted","Data":"e8cf75a19d830090f3ceecafe2c2eae809fda74d95e52e998b427ea739c2f404"}
Oct 30 13:03:43.121793 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:43.121768    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"
Oct 30 13:03:43.163053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:43.163004    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4" podStartSLOduration=3.162990447 podStartE2EDuration="3.162990447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:43.161463581 +0000 UTC m=+1611.685747813" watchObservedRunningTime="2026-10-30 13:03:43.162990447 +0000 UTC m=+1611.687274679"
Oct 30 13:03:43.362454 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:43.361829    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:03:43.362454 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:43.361888    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 13:03:43.573607 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:43.573569    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:03:43.573785 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:43.573622    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw" podUID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 13:03:43.643763 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:43.643722    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08" path="/var/lib/kubelet/pods/d1c8cedf-2ff2-4f5b-a00d-36c4b7e94f08/volumes"
Oct 30 13:03:44.361294 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:44.361261    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:03:44.361513 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:44.361311    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 13:03:44.573142 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:44.573105    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:03:44.573578 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:44.573158    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw" podUID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 13:03:45.147945 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:03:45.147888    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:03:45.362003 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:45.361974    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:03:45.362183 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:45.362027    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 13:03:45.573657 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:45.573614    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:03:45.574045 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:45.573668    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw" podUID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 13:03:45.722420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:45.722368    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:03:46.361220 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:46.361185    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:03:46.361433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:46.361240    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 13:03:46.573789 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:46.573740    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:03:46.574252 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:46.573810    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw" podUID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 13:03:47.366182 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:47.366152    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:03:47.367353 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:47.367328    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:03:47.578474 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:47.578446    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:03:47.578861 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:47.578797    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:03:47.579690 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:47.579670    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:03:49.468724 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:49.468690    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"]
Oct 30 13:03:49.469247 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:49.468947    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92" podUID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerName="istio-proxy" containerID="cri-o://ff087d509b01f5a47111f54afb25632beef37f689ff620ddc7dd936c34a5ad4f" gracePeriod=30
Oct 30 13:03:50.072430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.067463    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"]
Oct 30 13:03:50.076433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.076268    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"]
Oct 30 13:03:50.076433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.076388    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.098529 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.098492    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-workload-certs\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.098682 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.098544    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-istio-data\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.098682 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.098630    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612f5032-bf59-40f9-b043-e20395d385b2-istio-token\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.098813 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.098694    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-credential-socket\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.098869 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.098846    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612f5032-bf59-40f9-b043-e20395d385b2-istiod-ca-cert\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.098975 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.098957    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzdfj\" (UniqueName: \"kubernetes.io/projected/612f5032-bf59-40f9-b043-e20395d385b2-kube-api-access-mzdfj\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.099043 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.098991    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-istio-envoy\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.099102 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.099049    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-workload-socket\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.099102 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.099089    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612f5032-bf59-40f9-b043-e20395d385b2-istio-podinfo\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.168417 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.158794    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-f73c9f-openshift-default-dockercfg-zj7p2\""
Oct 30 13:03:50.202467 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.202428    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-workload-certs\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.203164 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.203139    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-istio-data\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.204736 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.203513    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612f5032-bf59-40f9-b043-e20395d385b2-istio-token\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.204736 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.203533    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-istio-data\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.204736 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.203072    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-workload-certs\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.204736 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.203568    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-credential-socket\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.204736 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.203683    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612f5032-bf59-40f9-b043-e20395d385b2-istiod-ca-cert\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.204736 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.203774    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzdfj\" (UniqueName: \"kubernetes.io/projected/612f5032-bf59-40f9-b043-e20395d385b2-kube-api-access-mzdfj\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.204736 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.203788    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-credential-socket\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.204736 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.203803    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-istio-envoy\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.204736 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.203855    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-workload-socket\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.204736 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.203893    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612f5032-bf59-40f9-b043-e20395d385b2-istio-podinfo\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.222896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.221969    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-workload-socket\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.222896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.222366    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612f5032-bf59-40f9-b043-e20395d385b2-istio-podinfo\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.223668 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.223638    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612f5032-bf59-40f9-b043-e20395d385b2-istiod-ca-cert\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.227247 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.225829    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612f5032-bf59-40f9-b043-e20395d385b2-istio-token\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.227247 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.226621    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzdfj\" (UniqueName: \"kubernetes.io/projected/612f5032-bf59-40f9-b043-e20395d385b2-kube-api-access-mzdfj\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.233480 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.229192    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-istio-envoy\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.402358 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.402271    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:50.773580 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.773551    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"]
Oct 30 13:03:50.774775 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:03:50.774746    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612f5032_bf59_40f9_b043_e20395d385b2.slice/crio-fe55e1b85f69522bc41ba99eccd2f74ca5003bb35d85949cd9fe0fd009251b2f WatchSource:0}: Error finding container fe55e1b85f69522bc41ba99eccd2f74ca5003bb35d85949cd9fe0fd009251b2f: Status 404 returned error can't find the container with id fe55e1b85f69522bc41ba99eccd2f74ca5003bb35d85949cd9fe0fd009251b2f
Oct 30 13:03:50.777601 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.777568    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:50.777724 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.777647    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:50.777724 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:50.777692    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:51.236281 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:51.236245    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" event={"ID":"612f5032-bf59-40f9-b043-e20395d385b2","Type":"ContainerStarted","Data":"1f685f1dee63842b48ade8a9418369c0122cade121a5ab5604ef91e408638800"}
Oct 30 13:03:51.236451 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:51.236303    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" event={"ID":"612f5032-bf59-40f9-b043-e20395d385b2","Type":"ContainerStarted","Data":"fe55e1b85f69522bc41ba99eccd2f74ca5003bb35d85949cd9fe0fd009251b2f"}
Oct 30 13:03:51.281748 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:51.281694    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" podStartSLOduration=2.281675249 podStartE2EDuration="2.281675249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:51.279294404 +0000 UTC m=+1619.803578622" watchObservedRunningTime="2026-10-30 13:03:51.281675249 +0000 UTC m=+1619.805959468"
Oct 30 13:03:51.405066 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:51.405033    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:51.425787 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:51.425714    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:52.243981 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:52.243953    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:52.244968 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:52.244949    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:03:54.139489 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.139456    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"
Oct 30 13:03:54.139893 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.139529    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"
Oct 30 13:03:54.719035 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.719000    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"]
Oct 30 13:03:54.726652 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.726608    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"
Oct 30 13:03:54.739189 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.739129    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"]
Oct 30 13:03:54.744813 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.744783    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:03:54.744940 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.744860    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:03:54.744940 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.744860    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:03:54.744940 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.744910    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:03:54.762681 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.762651    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:03:54.811287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.811101    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-envoy\") pod \"cee173cf-090c-42b7-8427-36e58cc4f3ba\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") "
Oct 30 13:03:54.811287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.811144    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-workload-socket\") pod \"cee173cf-090c-42b7-8427-36e58cc4f3ba\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") "
Oct 30 13:03:54.811287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.811246    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee173cf-090c-42b7-8427-36e58cc4f3ba-kube-api-access-wnzxt\" (UniqueName: \"kubernetes.io/projected/cee173cf-090c-42b7-8427-36e58cc4f3ba-kube-api-access-wnzxt\") pod \"cee173cf-090c-42b7-8427-36e58cc4f3ba\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") "
Oct 30 13:03:54.811287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.811278    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-token\" (UniqueName: \"kubernetes.io/projected/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-token\") pod \"cee173cf-090c-42b7-8427-36e58cc4f3ba\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") "
Oct 30 13:03:54.811663 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.811334    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-credential-socket\") pod \"cee173cf-090c-42b7-8427-36e58cc4f3ba\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") "
Oct 30 13:03:54.811663 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.811438    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee173cf-090c-42b7-8427-36e58cc4f3ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee173cf-090c-42b7-8427-36e58cc4f3ba-istiod-ca-cert\") pod \"cee173cf-090c-42b7-8427-36e58cc4f3ba\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") "
Oct 30 13:03:54.811663 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.811581    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-data\") pod \"cee173cf-090c-42b7-8427-36e58cc4f3ba\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") "
Oct 30 13:03:54.811663 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.811616    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-podinfo\") pod \"cee173cf-090c-42b7-8427-36e58cc4f3ba\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") "
Oct 30 13:03:54.811663 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.811655    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-workload-certs\") pod \"cee173cf-090c-42b7-8427-36e58cc4f3ba\" (UID: \"cee173cf-090c-42b7-8427-36e58cc4f3ba\") "
Oct 30 13:03:54.811998 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.811977    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80f0ba6d-338f-4516-9292-6947b0c7fe76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz\" (UID: \"80f0ba6d-338f-4516-9292-6947b0c7fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"
Oct 30 13:03:54.812077 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.812054    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80f0ba6d-338f-4516-9292-6947b0c7fe76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz\" (UID: \"80f0ba6d-338f-4516-9292-6947b0c7fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"
Oct 30 13:03:54.812210 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.812195    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jww87\" (UniqueName: \"kubernetes.io/projected/80f0ba6d-338f-4516-9292-6947b0c7fe76-kube-api-access-jww87\") pod \"maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz\" (UID: \"80f0ba6d-338f-4516-9292-6947b0c7fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"
Oct 30 13:03:54.813198 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.813169    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-data" pod "cee173cf-090c-42b7-8427-36e58cc4f3ba" (UID: "cee173cf-090c-42b7-8427-36e58cc4f3ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:03:54.813321 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.813207    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee173cf-090c-42b7-8427-36e58cc4f3ba-istiod-ca-cert" pod "cee173cf-090c-42b7-8427-36e58cc4f3ba" (UID: "cee173cf-090c-42b7-8427-36e58cc4f3ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:54.813483 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.813461    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-workload-socket" pod "cee173cf-090c-42b7-8427-36e58cc4f3ba" (UID: "cee173cf-090c-42b7-8427-36e58cc4f3ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:03:54.813696 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.813674    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-credential-socket" pod "cee173cf-090c-42b7-8427-36e58cc4f3ba" (UID: "cee173cf-090c-42b7-8427-36e58cc4f3ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:03:54.813766 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.813712    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-workload-certs" pod "cee173cf-090c-42b7-8427-36e58cc4f3ba" (UID: "cee173cf-090c-42b7-8427-36e58cc4f3ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:03:54.816429 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.816370    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-envoy" pod "cee173cf-090c-42b7-8427-36e58cc4f3ba" (UID: "cee173cf-090c-42b7-8427-36e58cc4f3ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:03:54.816689 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.816668    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-token" pod "cee173cf-090c-42b7-8427-36e58cc4f3ba" (UID: "cee173cf-090c-42b7-8427-36e58cc4f3ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:54.816833 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.816810    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-podinfo" pod "cee173cf-090c-42b7-8427-36e58cc4f3ba" (UID: "cee173cf-090c-42b7-8427-36e58cc4f3ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:03:54.817074 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.817044    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee173cf-090c-42b7-8427-36e58cc4f3ba-kube-api-access-wnzxt" pod "cee173cf-090c-42b7-8427-36e58cc4f3ba" (UID: "cee173cf-090c-42b7-8427-36e58cc4f3ba"). InnerVolumeSpecName "kube-api-access-wnzxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:54.913820 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.913788    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jww87\" (UniqueName: \"kubernetes.io/projected/80f0ba6d-338f-4516-9292-6947b0c7fe76-kube-api-access-jww87\") pod \"maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz\" (UID: \"80f0ba6d-338f-4516-9292-6947b0c7fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"
Oct 30 13:03:54.914011 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.913998    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80f0ba6d-338f-4516-9292-6947b0c7fe76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz\" (UID: \"80f0ba6d-338f-4516-9292-6947b0c7fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"
Oct 30 13:03:54.914071 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.914036    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80f0ba6d-338f-4516-9292-6947b0c7fe76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz\" (UID: \"80f0ba6d-338f-4516-9292-6947b0c7fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"
Oct 30 13:03:54.914153 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.914142    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:54.914214 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.914158    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:54.914214 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.914168    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:54.914214 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.914176    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:54.914214 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.914186    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:54.914214 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.914194    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnzxt\" (UniqueName: \"kubernetes.io/projected/cee173cf-090c-42b7-8427-36e58cc4f3ba-kube-api-access-wnzxt\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:54.914214 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.914202    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee173cf-090c-42b7-8427-36e58cc4f3ba-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:54.914214 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.914210    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee173cf-090c-42b7-8427-36e58cc4f3ba-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:54.914633 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.914219    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee173cf-090c-42b7-8427-36e58cc4f3ba-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:54.914792 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.914775    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80f0ba6d-338f-4516-9292-6947b0c7fe76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz\" (UID: \"80f0ba6d-338f-4516-9292-6947b0c7fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"
Oct 30 13:03:54.916590 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.916565    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80f0ba6d-338f-4516-9292-6947b0c7fe76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz\" (UID: \"80f0ba6d-338f-4516-9292-6947b0c7fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"
Oct 30 13:03:54.932898 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:54.932878    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jww87\" (UniqueName: \"kubernetes.io/projected/80f0ba6d-338f-4516-9292-6947b0c7fe76-kube-api-access-jww87\") pod \"maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz\" (UID: \"80f0ba6d-338f-4516-9292-6947b0c7fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"
Oct 30 13:03:55.076204 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.076126    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"
Oct 30 13:03:55.272388 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.272351    5837 generic.go:356] "Generic (PLEG): container finished" podID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerID="ff087d509b01f5a47111f54afb25632beef37f689ff620ddc7dd936c34a5ad4f" exitCode=0
Oct 30 13:03:55.273143 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.272672    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92" event={"ID":"cee173cf-090c-42b7-8427-36e58cc4f3ba","Type":"ContainerDied","Data":"ff087d509b01f5a47111f54afb25632beef37f689ff620ddc7dd936c34a5ad4f"}
Oct 30 13:03:55.273143 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.272746    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92" event={"ID":"cee173cf-090c-42b7-8427-36e58cc4f3ba","Type":"ContainerDied","Data":"943c2fb08464fa07d2335a034a9499e275baafb602f766b0a94fad050f6bf639"}
Oct 30 13:03:55.273143 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.272775    5837 scope.go:122] "RemoveContainer" containerID="ff087d509b01f5a47111f54afb25632beef37f689ff620ddc7dd936c34a5ad4f"
Oct 30 13:03:55.273143 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.272993    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"
Oct 30 13:03:55.287896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.287878    5837 scope.go:122] "RemoveContainer" containerID="cc2a500ec16ebe534e04419d11c56dd3f5eb9f695c2782752a8f4390b4ca697f"
Oct 30 13:03:55.298140 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.298123    5837 scope.go:122] "RemoveContainer" containerID="ff087d509b01f5a47111f54afb25632beef37f689ff620ddc7dd936c34a5ad4f"
Oct 30 13:03:55.298456 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:03:55.298439    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff087d509b01f5a47111f54afb25632beef37f689ff620ddc7dd936c34a5ad4f\": container with ID starting with ff087d509b01f5a47111f54afb25632beef37f689ff620ddc7dd936c34a5ad4f not found: ID does not exist" containerID="ff087d509b01f5a47111f54afb25632beef37f689ff620ddc7dd936c34a5ad4f"
Oct 30 13:03:55.298507 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.298464    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff087d509b01f5a47111f54afb25632beef37f689ff620ddc7dd936c34a5ad4f"} err="failed to get container status \"ff087d509b01f5a47111f54afb25632beef37f689ff620ddc7dd936c34a5ad4f\": rpc error: code = NotFound desc = could not find container \"ff087d509b01f5a47111f54afb25632beef37f689ff620ddc7dd936c34a5ad4f\": container with ID starting with ff087d509b01f5a47111f54afb25632beef37f689ff620ddc7dd936c34a5ad4f not found: ID does not exist"
Oct 30 13:03:55.298507 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.298480    5837 scope.go:122] "RemoveContainer" containerID="cc2a500ec16ebe534e04419d11c56dd3f5eb9f695c2782752a8f4390b4ca697f"
Oct 30 13:03:55.298735 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:03:55.298711    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2a500ec16ebe534e04419d11c56dd3f5eb9f695c2782752a8f4390b4ca697f\": container with ID starting with cc2a500ec16ebe534e04419d11c56dd3f5eb9f695c2782752a8f4390b4ca697f not found: ID does not exist" containerID="cc2a500ec16ebe534e04419d11c56dd3f5eb9f695c2782752a8f4390b4ca697f"
Oct 30 13:03:55.298772 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.298745    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2a500ec16ebe534e04419d11c56dd3f5eb9f695c2782752a8f4390b4ca697f"} err="failed to get container status \"cc2a500ec16ebe534e04419d11c56dd3f5eb9f695c2782752a8f4390b4ca697f\": rpc error: code = NotFound desc = could not find container \"cc2a500ec16ebe534e04419d11c56dd3f5eb9f695c2782752a8f4390b4ca697f\": container with ID starting with cc2a500ec16ebe534e04419d11c56dd3f5eb9f695c2782752a8f4390b4ca697f not found: ID does not exist"
Oct 30 13:03:55.335621 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.335555    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"]
Oct 30 13:03:55.339063 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.339035    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-87f1b5-openshift-default-649cd9d869-bgt92"]
Oct 30 13:03:55.376675 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:03:55.376649    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f0ba6d_338f_4516_9292_6947b0c7fe76.slice/crio-78fe1e80dbdd820c1e523075571aed39c09efeedb9ac22179c693e73c7b2c38a WatchSource:0}: Error finding container 78fe1e80dbdd820c1e523075571aed39c09efeedb9ac22179c693e73c7b2c38a: Status 404 returned error can't find the container with id 78fe1e80dbdd820c1e523075571aed39c09efeedb9ac22179c693e73c7b2c38a
Oct 30 13:03:55.376915 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.376895    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"]
Oct 30 13:03:55.607281 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.607193    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee173cf-090c-42b7-8427-36e58cc4f3ba" path="/var/lib/kubelet/pods/cee173cf-090c-42b7-8427-36e58cc4f3ba/volumes"
Oct 30 13:03:55.657317 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.657289    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"]
Oct 30 13:03:55.658030 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.658014    5837 cpu_manager.go:414] "containerMap: removing container" podUID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerName="istio-proxy"
Oct 30 13:03:55.658085 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.658032    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerName="istio-proxy"
Oct 30 13:03:55.658085 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.658061    5837 cpu_manager.go:414] "containerMap: removing container" podUID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerName="istio-proxy"
Oct 30 13:03:55.658085 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.658066    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerName="istio-proxy"
Oct 30 13:03:55.658183 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.658169    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerName="istio-proxy"
Oct 30 13:03:55.661923 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.661905    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"
Oct 30 13:03:55.678728 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.678703    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f73c9f-dockercfg-jqqhq\""
Oct 30 13:03:55.678728 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.678707    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f73c9f\""
Oct 30 13:03:55.680498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.680478    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"]
Oct 30 13:03:55.757420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.756876    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5109255-b08f-4f6e-a83d-325b173f11be-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq\" (UID: \"f5109255-b08f-4f6e-a83d-325b173f11be\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"
Oct 30 13:03:55.757420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.756976    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjxkg\" (UniqueName: \"kubernetes.io/projected/f5109255-b08f-4f6e-a83d-325b173f11be-kube-api-access-pjxkg\") pod \"payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq\" (UID: \"f5109255-b08f-4f6e-a83d-325b173f11be\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"
Oct 30 13:03:55.891211 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.887134    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjxkg\" (UniqueName: \"kubernetes.io/projected/f5109255-b08f-4f6e-a83d-325b173f11be-kube-api-access-pjxkg\") pod \"payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq\" (UID: \"f5109255-b08f-4f6e-a83d-325b173f11be\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"
Oct 30 13:03:55.891211 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.887630    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5109255-b08f-4f6e-a83d-325b173f11be-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq\" (UID: \"f5109255-b08f-4f6e-a83d-325b173f11be\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"
Oct 30 13:03:55.914420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.911025    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"]
Oct 30 13:03:55.914420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.912383    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee173cf-090c-42b7-8427-36e58cc4f3ba" containerName="istio-proxy"
Oct 30 13:03:55.914420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.913598    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5109255-b08f-4f6e-a83d-325b173f11be-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq\" (UID: \"f5109255-b08f-4f6e-a83d-325b173f11be\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"
Oct 30 13:03:55.925484 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.924378    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"]
Oct 30 13:03:55.925484 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.924523    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"
Oct 30 13:03:55.948593 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.934830    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjxkg\" (UniqueName: \"kubernetes.io/projected/f5109255-b08f-4f6e-a83d-325b173f11be-kube-api-access-pjxkg\") pod \"payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq\" (UID: \"f5109255-b08f-4f6e-a83d-325b173f11be\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"
Oct 30 13:03:55.976815 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:55.976780    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"
Oct 30 13:03:56.096728 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:56.096686    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb1a7e8-4931-4264-bc9d-194832b35e35-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th\" (UID: \"bbb1a7e8-4931-4264-bc9d-194832b35e35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"
Oct 30 13:03:56.096902 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:56.096883    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhht7\" (UniqueName: \"kubernetes.io/projected/bbb1a7e8-4931-4264-bc9d-194832b35e35-kube-api-access-hhht7\") pod \"payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th\" (UID: \"bbb1a7e8-4931-4264-bc9d-194832b35e35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"
Oct 30 13:03:56.198200 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:56.198121    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhht7\" (UniqueName: \"kubernetes.io/projected/bbb1a7e8-4931-4264-bc9d-194832b35e35-kube-api-access-hhht7\") pod \"payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th\" (UID: \"bbb1a7e8-4931-4264-bc9d-194832b35e35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"
Oct 30 13:03:56.198405 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:56.198375    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb1a7e8-4931-4264-bc9d-194832b35e35-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th\" (UID: \"bbb1a7e8-4931-4264-bc9d-194832b35e35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"
Oct 30 13:03:56.199115 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:56.199095    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb1a7e8-4931-4264-bc9d-194832b35e35-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th\" (UID: \"bbb1a7e8-4931-4264-bc9d-194832b35e35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"
Oct 30 13:03:56.217973 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:56.217950    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhht7\" (UniqueName: \"kubernetes.io/projected/bbb1a7e8-4931-4264-bc9d-194832b35e35-kube-api-access-hhht7\") pod \"payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th\" (UID: \"bbb1a7e8-4931-4264-bc9d-194832b35e35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"
Oct 30 13:03:56.282804 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:56.282769    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz" event={"ID":"80f0ba6d-338f-4516-9292-6947b0c7fe76","Type":"ContainerStarted","Data":"151e60e39d83fd2506c45e4d5739e742297a2df0e00982242665971576982106"}
Oct 30 13:03:56.283223 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:56.283000    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz" event={"ID":"80f0ba6d-338f-4516-9292-6947b0c7fe76","Type":"ContainerStarted","Data":"78fe1e80dbdd820c1e523075571aed39c09efeedb9ac22179c693e73c7b2c38a"}
Oct 30 13:03:56.299062 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:56.299036    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"
Oct 30 13:03:56.322066 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:56.322043    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"]
Oct 30 13:03:56.324780 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:03:56.324753    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5109255_b08f_4f6e_a83d_325b173f11be.slice/crio-78a2c37aaa5cd10e5c828a6dc7887af42d7de7d5f49b98d9be193a6eda394d43 WatchSource:0}: Error finding container 78a2c37aaa5cd10e5c828a6dc7887af42d7de7d5f49b98d9be193a6eda394d43: Status 404 returned error can't find the container with id 78a2c37aaa5cd10e5c828a6dc7887af42d7de7d5f49b98d9be193a6eda394d43
Oct 30 13:03:56.330153 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:56.330104    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz" podStartSLOduration=2.330091191 podStartE2EDuration="2.330091191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:56.327590641 +0000 UTC m=+1624.851874861" watchObservedRunningTime="2026-10-30 13:03:56.330091191 +0000 UTC m=+1624.854375409"
Oct 30 13:03:56.602510 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:56.602485    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"]
Oct 30 13:03:56.667401 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:03:56.667371    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb1a7e8_4931_4264_bc9d_194832b35e35.slice/crio-fa36e4cac95fcc32fb79e06f28715704c8808db9608ed7e1d41c2209a73ca5a1 WatchSource:0}: Error finding container fa36e4cac95fcc32fb79e06f28715704c8808db9608ed7e1d41c2209a73ca5a1: Status 404 returned error can't find the container with id fa36e4cac95fcc32fb79e06f28715704c8808db9608ed7e1d41c2209a73ca5a1
Oct 30 13:03:57.063856 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:57.063824    5837 scope.go:122] "RemoveContainer" containerID="ab04d4e2c580929e587a093ae9c6ebf55228db9baed18853997fcfa7ec82080f"
Oct 30 13:03:57.074619 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:57.074591    5837 scope.go:122] "RemoveContainer" containerID="d0b406065c330d5714b9226078467123796654a73d412ce7c231d4360b225f2c"
Oct 30 13:03:57.084424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:57.084377    5837 scope.go:122] "RemoveContainer" containerID="1682ee503a9a3203fecb921c6d69b9d075233cc3e919b7b2d9c48d55a2a9ad63"
Oct 30 13:03:57.098230 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:57.098205    5837 scope.go:122] "RemoveContainer" containerID="256da51b98f9c2df20b146c072269612a8df08b28524461a25c18c840a6ca161"
Oct 30 13:03:57.293608 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:57.293574    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq" event={"ID":"f5109255-b08f-4f6e-a83d-325b173f11be","Type":"ContainerStarted","Data":"9208d448a8b595d53369031b640abe967ef2e998f844bb8904d5c6d5660ced02"}
Oct 30 13:03:57.294014 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:57.293633    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq" event={"ID":"f5109255-b08f-4f6e-a83d-325b173f11be","Type":"ContainerStarted","Data":"78a2c37aaa5cd10e5c828a6dc7887af42d7de7d5f49b98d9be193a6eda394d43"}
Oct 30 13:03:57.294014 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:57.293663    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"
Oct 30 13:03:57.295380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:57.295351    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th" event={"ID":"bbb1a7e8-4931-4264-bc9d-194832b35e35","Type":"ContainerStarted","Data":"1e685b86b9d37e52173e6e1beda62d1985a06d025e949514a627a4889804c212"}
Oct 30 13:03:57.295555 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:57.295530    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th" event={"ID":"bbb1a7e8-4931-4264-bc9d-194832b35e35","Type":"ContainerStarted","Data":"fa36e4cac95fcc32fb79e06f28715704c8808db9608ed7e1d41c2209a73ca5a1"}
Oct 30 13:03:57.295609 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:57.295576    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"
Oct 30 13:03:57.296913 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:57.296891    5837 generic.go:356] "Generic (PLEG): container finished" podID="80f0ba6d-338f-4516-9292-6947b0c7fe76" containerID="151e60e39d83fd2506c45e4d5739e742297a2df0e00982242665971576982106" exitCode=0
Oct 30 13:03:57.297011 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:57.296937    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz" event={"ID":"80f0ba6d-338f-4516-9292-6947b0c7fe76","Type":"ContainerDied","Data":"151e60e39d83fd2506c45e4d5739e742297a2df0e00982242665971576982106"}
Oct 30 13:03:57.333587 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:57.333509    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq" podStartSLOduration=2.333495201 podStartE2EDuration="2.333495201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:57.331445625 +0000 UTC m=+1625.855729857" watchObservedRunningTime="2026-10-30 13:03:57.333495201 +0000 UTC m=+1625.857779421"
Oct 30 13:03:57.371577 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:57.371522    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th" podStartSLOduration=2.371508519 podStartE2EDuration="2.371508519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:57.368241662 +0000 UTC m=+1625.892525884" watchObservedRunningTime="2026-10-30 13:03:57.371508519 +0000 UTC m=+1625.895792737"
Oct 30 13:03:58.481378 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:58.481352    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"
Oct 30 13:03:58.635921 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:58.635842    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80f0ba6d-338f-4516-9292-6947b0c7fe76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80f0ba6d-338f-4516-9292-6947b0c7fe76-maas-api-xks-ca\") pod \"80f0ba6d-338f-4516-9292-6947b0c7fe76\" (UID: \"80f0ba6d-338f-4516-9292-6947b0c7fe76\") "
Oct 30 13:03:58.635921 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:58.635914    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f0ba6d-338f-4516-9292-6947b0c7fe76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80f0ba6d-338f-4516-9292-6947b0c7fe76-maas-api-openshift-ca\") pod \"80f0ba6d-338f-4516-9292-6947b0c7fe76\" (UID: \"80f0ba6d-338f-4516-9292-6947b0c7fe76\") "
Oct 30 13:03:58.636162 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:58.636045    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f0ba6d-338f-4516-9292-6947b0c7fe76-kube-api-access-jww87\" (UniqueName: \"kubernetes.io/projected/80f0ba6d-338f-4516-9292-6947b0c7fe76-kube-api-access-jww87\") pod \"80f0ba6d-338f-4516-9292-6947b0c7fe76\" (UID: \"80f0ba6d-338f-4516-9292-6947b0c7fe76\") "
Oct 30 13:03:58.636307 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:58.636280    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f0ba6d-338f-4516-9292-6947b0c7fe76-maas-api-openshift-ca" pod "80f0ba6d-338f-4516-9292-6947b0c7fe76" (UID: "80f0ba6d-338f-4516-9292-6947b0c7fe76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:58.636758 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:58.636705    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80f0ba6d-338f-4516-9292-6947b0c7fe76-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:58.638209 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:58.638185    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f0ba6d-338f-4516-9292-6947b0c7fe76-maas-api-xks-ca" pod "80f0ba6d-338f-4516-9292-6947b0c7fe76" (UID: "80f0ba6d-338f-4516-9292-6947b0c7fe76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:03:58.638442 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:58.638416    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f0ba6d-338f-4516-9292-6947b0c7fe76-kube-api-access-jww87" pod "80f0ba6d-338f-4516-9292-6947b0c7fe76" (UID: "80f0ba6d-338f-4516-9292-6947b0c7fe76"). InnerVolumeSpecName "kube-api-access-jww87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:58.738217 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:58.738177    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80f0ba6d-338f-4516-9292-6947b0c7fe76-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:58.738217 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:58.738208    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jww87\" (UniqueName: \"kubernetes.io/projected/80f0ba6d-338f-4516-9292-6947b0c7fe76-kube-api-access-jww87\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:03:59.313501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:59.313469    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz" event={"ID":"80f0ba6d-338f-4516-9292-6947b0c7fe76","Type":"ContainerDied","Data":"78fe1e80dbdd820c1e523075571aed39c09efeedb9ac22179c693e73c7b2c38a"}
Oct 30 13:03:59.313677 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:59.313504    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"
Oct 30 13:03:59.313677 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:03:59.313518    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78fe1e80dbdd820c1e523075571aed39c09efeedb9ac22179c693e73c7b2c38a"
Oct 30 13:04:05.157577 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:05.157494    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"]
Oct 30 13:04:05.190802 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:05.190776    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"]
Oct 30 13:04:08.306948 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:08.306922    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"
Oct 30 13:04:08.307334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:08.306984    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"
Oct 30 13:04:10.158453 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.158360    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx" podUID="a34067f6-150e-46ca-b023-912b4c89c124" containerName="payload-processing" containerID="cri-o://88dd3b734b33ef440e356af69711889498e96f0f50951c68d0dfd88e8fd26a1a" gracePeriod=25
Oct 30 13:04:10.191884 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.191853    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd" podUID="3ebbba38-3e78-485f-abc4-fda302f2888e" containerName="payload-pre-processing" containerID="cri-o://a6674a10a8a91507cfc773111b7b2332032a1e2b6a6f1567a849bb26e8f0e516" gracePeriod=25
Oct 30 13:04:10.422148 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.416370    5837 generic.go:356] "Generic (PLEG): container finished" podID="3ebbba38-3e78-485f-abc4-fda302f2888e" containerID="a6674a10a8a91507cfc773111b7b2332032a1e2b6a6f1567a849bb26e8f0e516" exitCode=0
Oct 30 13:04:10.422148 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.416512    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd" event={"ID":"3ebbba38-3e78-485f-abc4-fda302f2888e","Type":"ContainerDied","Data":"a6674a10a8a91507cfc773111b7b2332032a1e2b6a6f1567a849bb26e8f0e516"}
Oct 30 13:04:10.422148 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.418373    5837 generic.go:356] "Generic (PLEG): container finished" podID="a34067f6-150e-46ca-b023-912b4c89c124" containerID="88dd3b734b33ef440e356af69711889498e96f0f50951c68d0dfd88e8fd26a1a" exitCode=0
Oct 30 13:04:10.422148 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.418507    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx" event={"ID":"a34067f6-150e-46ca-b023-912b4c89c124","Type":"ContainerDied","Data":"88dd3b734b33ef440e356af69711889498e96f0f50951c68d0dfd88e8fd26a1a"}
Oct 30 13:04:10.523456 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.523408    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"
Oct 30 13:04:10.558075 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.558049    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"
Oct 30 13:04:10.684843 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.684768    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a34067f6-150e-46ca-b023-912b4c89c124-kube-api-access-f8cth\" (UniqueName: \"kubernetes.io/projected/a34067f6-150e-46ca-b023-912b4c89c124-kube-api-access-f8cth\") pod \"a34067f6-150e-46ca-b023-912b4c89c124\" (UID: \"a34067f6-150e-46ca-b023-912b4c89c124\") "
Oct 30 13:04:10.685000 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.684984    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a34067f6-150e-46ca-b023-912b4c89c124-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34067f6-150e-46ca-b023-912b4c89c124-plugins-config-volume\") pod \"a34067f6-150e-46ca-b023-912b4c89c124\" (UID: \"a34067f6-150e-46ca-b023-912b4c89c124\") "
Oct 30 13:04:10.685053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.685003    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ebbba38-3e78-485f-abc4-fda302f2888e-kube-api-access-mprng\" (UniqueName: \"kubernetes.io/projected/3ebbba38-3e78-485f-abc4-fda302f2888e-kube-api-access-mprng\") pod \"3ebbba38-3e78-485f-abc4-fda302f2888e\" (UID: \"3ebbba38-3e78-485f-abc4-fda302f2888e\") "
Oct 30 13:04:10.685110 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.685056    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ebbba38-3e78-485f-abc4-fda302f2888e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ebbba38-3e78-485f-abc4-fda302f2888e-plugins-config-volume\") pod \"3ebbba38-3e78-485f-abc4-fda302f2888e\" (UID: \"3ebbba38-3e78-485f-abc4-fda302f2888e\") "
Oct 30 13:04:10.685478 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.685448    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34067f6-150e-46ca-b023-912b4c89c124-plugins-config-volume" pod "a34067f6-150e-46ca-b023-912b4c89c124" (UID: "a34067f6-150e-46ca-b023-912b4c89c124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:10.685608 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.685493    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ebbba38-3e78-485f-abc4-fda302f2888e-plugins-config-volume" pod "3ebbba38-3e78-485f-abc4-fda302f2888e" (UID: "3ebbba38-3e78-485f-abc4-fda302f2888e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:10.685848 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.685828    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34067f6-150e-46ca-b023-912b4c89c124-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:10.685890 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.685857    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ebbba38-3e78-485f-abc4-fda302f2888e-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:10.687134 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.687117    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34067f6-150e-46ca-b023-912b4c89c124-kube-api-access-f8cth" pod "a34067f6-150e-46ca-b023-912b4c89c124" (UID: "a34067f6-150e-46ca-b023-912b4c89c124"). InnerVolumeSpecName "kube-api-access-f8cth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:10.687201 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.687152    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebbba38-3e78-485f-abc4-fda302f2888e-kube-api-access-mprng" pod "3ebbba38-3e78-485f-abc4-fda302f2888e" (UID: "3ebbba38-3e78-485f-abc4-fda302f2888e"). InnerVolumeSpecName "kube-api-access-mprng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:10.787658 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.787633    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mprng\" (UniqueName: \"kubernetes.io/projected/3ebbba38-3e78-485f-abc4-fda302f2888e-kube-api-access-mprng\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:10.787658 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:10.787658    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8cth\" (UniqueName: \"kubernetes.io/projected/a34067f6-150e-46ca-b023-912b4c89c124-kube-api-access-f8cth\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:11.426848 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:11.426760    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"
Oct 30 13:04:11.426848 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:11.426770    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd" event={"ID":"3ebbba38-3e78-485f-abc4-fda302f2888e","Type":"ContainerDied","Data":"bd587c620ffb694b5c405a2ee857a972fbb08ce0510682f3dec2d7c117388da0"}
Oct 30 13:04:11.426848 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:11.426826    5837 scope.go:122] "RemoveContainer" containerID="a6674a10a8a91507cfc773111b7b2332032a1e2b6a6f1567a849bb26e8f0e516"
Oct 30 13:04:11.428666 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:11.428643    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"
Oct 30 13:04:11.428790 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:11.428665    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx" event={"ID":"a34067f6-150e-46ca-b023-912b4c89c124","Type":"ContainerDied","Data":"91971036180c34eb87ae043a967691ba5a4e30bf323ade72457fa8ca5ea5dfd2"}
Oct 30 13:04:11.439488 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:11.439470    5837 scope.go:122] "RemoveContainer" containerID="88dd3b734b33ef440e356af69711889498e96f0f50951c68d0dfd88e8fd26a1a"
Oct 30 13:04:11.480380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:11.480356    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"]
Oct 30 13:04:11.483527 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:11.483510    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2382a9-6578bcfcf5-rd7zd"]
Oct 30 13:04:11.525578 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:11.525540    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"]
Oct 30 13:04:11.530988 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:11.530962    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2382a9-8594fcccfb-74rlx"]
Oct 30 13:04:11.606459 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:11.606433    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ebbba38-3e78-485f-abc4-fda302f2888e" path="/var/lib/kubelet/pods/3ebbba38-3e78-485f-abc4-fda302f2888e/volumes"
Oct 30 13:04:11.606925 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:11.606904    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a34067f6-150e-46ca-b023-912b4c89c124" path="/var/lib/kubelet/pods/a34067f6-150e-46ca-b023-912b4c89c124/volumes"
Oct 30 13:04:15.097049 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:04:15.097007    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:04:20.939216 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:20.939179    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"]
Oct 30 13:04:20.950656 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:20.950627    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2382a9-2zdvz"]
Oct 30 13:04:21.606266 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:21.606232    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f0ba6d-338f-4516-9292-6947b0c7fe76" path="/var/lib/kubelet/pods/80f0ba6d-338f-4516-9292-6947b0c7fe76/volumes"
Oct 30 13:04:28.709106 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:28.709066    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"]
Oct 30 13:04:28.709633 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:28.709304    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerName="istio-proxy" containerID="cri-o://23bffeb0690db510e246651f4b333fec203c44b99fba02d4f29b7bf6feab5d1b" gracePeriod=30
Oct 30 13:04:32.368920 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:32.368886    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:04:32.369296 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:32.368953    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:04:33.738694 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.738659    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"]
Oct 30 13:04:33.739620 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.739598    5837 cpu_manager.go:414] "containerMap: removing container" podUID="3ebbba38-3e78-485f-abc4-fda302f2888e" containerName="payload-pre-processing"
Oct 30 13:04:33.739620 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.739619    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ebbba38-3e78-485f-abc4-fda302f2888e" containerName="payload-pre-processing"
Oct 30 13:04:33.739848 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.739666    5837 cpu_manager.go:414] "containerMap: removing container" podUID="a34067f6-150e-46ca-b023-912b4c89c124" containerName="payload-processing"
Oct 30 13:04:33.739848 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.739674    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a34067f6-150e-46ca-b023-912b4c89c124" containerName="payload-processing"
Oct 30 13:04:33.739848 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.739705    5837 cpu_manager.go:414] "containerMap: removing container" podUID="80f0ba6d-338f-4516-9292-6947b0c7fe76" containerName="revoke-keys"
Oct 30 13:04:33.739848 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.739713    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f0ba6d-338f-4516-9292-6947b0c7fe76" containerName="revoke-keys"
Oct 30 13:04:33.740057 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.739933    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="a34067f6-150e-46ca-b023-912b4c89c124" containerName="payload-processing"
Oct 30 13:04:33.740057 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.739952    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f0ba6d-338f-4516-9292-6947b0c7fe76" containerName="revoke-keys"
Oct 30 13:04:33.740057 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.739973    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ebbba38-3e78-485f-abc4-fda302f2888e" containerName="payload-pre-processing"
Oct 30 13:04:33.744656 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.744547    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"
Oct 30 13:04:33.762917 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.762856    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"]
Oct 30 13:04:33.771027 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.770298    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:04:33.771027 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.770409    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:04:33.771027 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.770968    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:04:33.771351 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.771329    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:04:33.823473 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.823437    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh9tb\" (UniqueName: \"kubernetes.io/projected/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-kube-api-access-bh9tb\") pod \"maas-api-revoke-sub-39faa139-jnjpg\" (UID: \"9f22bb1d-f0c7-436e-b0d2-810e9098e80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"
Oct 30 13:04:33.824013 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.823989    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39faa139-jnjpg\" (UID: \"9f22bb1d-f0c7-436e-b0d2-810e9098e80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"
Oct 30 13:04:33.824326 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.824307    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39faa139-jnjpg\" (UID: \"9f22bb1d-f0c7-436e-b0d2-810e9098e80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"
Oct 30 13:04:33.937417 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.929502    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39faa139-jnjpg\" (UID: \"9f22bb1d-f0c7-436e-b0d2-810e9098e80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"
Oct 30 13:04:33.937417 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.929658    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh9tb\" (UniqueName: \"kubernetes.io/projected/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-kube-api-access-bh9tb\") pod \"maas-api-revoke-sub-39faa139-jnjpg\" (UID: \"9f22bb1d-f0c7-436e-b0d2-810e9098e80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"
Oct 30 13:04:33.937417 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.929852    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39faa139-jnjpg\" (UID: \"9f22bb1d-f0c7-436e-b0d2-810e9098e80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"
Oct 30 13:04:33.938561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.938532    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39faa139-jnjpg\" (UID: \"9f22bb1d-f0c7-436e-b0d2-810e9098e80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"
Oct 30 13:04:33.960789 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.960759    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39faa139-jnjpg\" (UID: \"9f22bb1d-f0c7-436e-b0d2-810e9098e80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"
Oct 30 13:04:33.976060 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:33.976030    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh9tb\" (UniqueName: \"kubernetes.io/projected/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-kube-api-access-bh9tb\") pod \"maas-api-revoke-sub-39faa139-jnjpg\" (UID: \"9f22bb1d-f0c7-436e-b0d2-810e9098e80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"
Oct 30 13:04:34.118491 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:34.118335    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"
Oct 30 13:04:34.445921 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:34.445880    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"]
Oct 30 13:04:34.821307 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:34.821257    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg" event={"ID":"9f22bb1d-f0c7-436e-b0d2-810e9098e80b","Type":"ContainerStarted","Data":"16e4de8ac37be6657e5aa33c4aef2afd3fc15badc5580fd33b7b3be29fa2a224"}
Oct 30 13:04:34.884685 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:34.864044    5837 generic.go:356] "Generic (PLEG): container finished" podID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerID="23bffeb0690db510e246651f4b333fec203c44b99fba02d4f29b7bf6feab5d1b" exitCode=0
Oct 30 13:04:34.884685 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:34.864226    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" event={"ID":"9e2ab740-6bfa-438b-a878-e8a852a9b1a1","Type":"ContainerDied","Data":"23bffeb0690db510e246651f4b333fec203c44b99fba02d4f29b7bf6feab5d1b"}
Oct 30 13:04:34.884685 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:34.864291    5837 scope.go:122] "RemoveContainer" containerID="ff6b665d8257d32f953ec76d48bf420e6fff2572de455306840fc3cb4bed198a"
Oct 30 13:04:35.178355 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.178303    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:04:35.297470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.283369    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-token\" (UniqueName: \"kubernetes.io/projected/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-token\") pod \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") "
Oct 30 13:04:35.297470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.283951    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-workload-socket\") pod \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") "
Oct 30 13:04:35.297470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.284238    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-workload-socket" pod "9e2ab740-6bfa-438b-a878-e8a852a9b1a1" (UID: "9e2ab740-6bfa-438b-a878-e8a852a9b1a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:35.297470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.284311    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-envoy\") pod \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") "
Oct 30 13:04:35.297470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.284689    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-workload-certs\") pod \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") "
Oct 30 13:04:35.297470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.284935    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-workload-certs" pod "9e2ab740-6bfa-438b-a878-e8a852a9b1a1" (UID: "9e2ab740-6bfa-438b-a878-e8a852a9b1a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:35.297470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.285031    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-data\") pod \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") "
Oct 30 13:04:35.297470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.285120    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-credential-socket\") pod \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") "
Oct 30 13:04:35.297470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.285164    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-podinfo\") pod \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") "
Oct 30 13:04:35.297470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.285326    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istiod-ca-cert\") pod \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") "
Oct 30 13:04:35.297470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.293605    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-credential-socket" pod "9e2ab740-6bfa-438b-a878-e8a852a9b1a1" (UID: "9e2ab740-6bfa-438b-a878-e8a852a9b1a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:35.297470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.293745    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-data" pod "9e2ab740-6bfa-438b-a878-e8a852a9b1a1" (UID: "9e2ab740-6bfa-438b-a878-e8a852a9b1a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:35.297470 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.294386    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istiod-ca-cert" pod "9e2ab740-6bfa-438b-a878-e8a852a9b1a1" (UID: "9e2ab740-6bfa-438b-a878-e8a852a9b1a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:35.330504 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.328756    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-kube-api-access-t467b" pod "9e2ab740-6bfa-438b-a878-e8a852a9b1a1" (UID: "9e2ab740-6bfa-438b-a878-e8a852a9b1a1"). InnerVolumeSpecName "kube-api-access-t467b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:35.330504 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.328868    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-token" pod "9e2ab740-6bfa-438b-a878-e8a852a9b1a1" (UID: "9e2ab740-6bfa-438b-a878-e8a852a9b1a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:35.368628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.349051    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-podinfo" pod "9e2ab740-6bfa-438b-a878-e8a852a9b1a1" (UID: "9e2ab740-6bfa-438b-a878-e8a852a9b1a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:04:35.574433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.514644    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-kube-api-access-t467b\" (UniqueName: \"kubernetes.io/projected/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-kube-api-access-t467b\") pod \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\" (UID: \"9e2ab740-6bfa-438b-a878-e8a852a9b1a1\") "
Oct 30 13:04:35.574433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.515538    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:35.574433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.515570    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t467b\" (UniqueName: \"kubernetes.io/projected/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-kube-api-access-t467b\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:35.574433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.515588    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:35.574433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.515603    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:35.574433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.515616    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:35.574433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.515629    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:35.574433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.515641    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:35.574433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.515655    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:35.606870 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.606673    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-envoy" pod "9e2ab740-6bfa-438b-a878-e8a852a9b1a1" (UID: "9e2ab740-6bfa-438b-a878-e8a852a9b1a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:35.622436 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.622408    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2ab740-6bfa-438b-a878-e8a852a9b1a1-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:35.954427 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.949173    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg" event={"ID":"9f22bb1d-f0c7-436e-b0d2-810e9098e80b","Type":"ContainerStarted","Data":"bc133416c9a9331b05457c01458d0ed8bc908e876db3cfb55f96e3d2b4b66631"}
Oct 30 13:04:35.969066 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.969020    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v" event={"ID":"9e2ab740-6bfa-438b-a878-e8a852a9b1a1","Type":"ContainerDied","Data":"e2c2c311839860178bdd2703e61f76d615c4ccb7437420cfe2e8a4e59ff95712"}
Oct 30 13:04:35.969379 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.969342    5837 scope.go:122] "RemoveContainer" containerID="23bffeb0690db510e246651f4b333fec203c44b99fba02d4f29b7bf6feab5d1b"
Oct 30 13:04:35.974838 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:35.973786    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"
Oct 30 13:04:36.000779 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.000725    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg" podStartSLOduration=3.0006913 podStartE2EDuration="3.0006913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:35.99978738 +0000 UTC m=+1664.524071607" watchObservedRunningTime="2026-10-30 13:04:36.0006913 +0000 UTC m=+1664.524975529"
Oct 30 13:04:36.088512 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.088443    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"]
Oct 30 13:04:36.122418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.107524    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-2382a9-openshift-default-65f4d6457-fr92v"]
Oct 30 13:04:36.321420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.317614    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"]
Oct 30 13:04:36.321420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.318767    5837 cpu_manager.go:414] "containerMap: removing container" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerName="istio-proxy"
Oct 30 13:04:36.321420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.318791    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerName="istio-proxy"
Oct 30 13:04:36.321420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.318838    5837 cpu_manager.go:414] "containerMap: removing container" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerName="istio-proxy"
Oct 30 13:04:36.321420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.318846    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerName="istio-proxy"
Oct 30 13:04:36.321420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.319054    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerName="istio-proxy"
Oct 30 13:04:36.517421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.515039    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"]
Oct 30 13:04:36.517421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.515176    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"
Oct 30 13:04:36.621446 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.591476    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52xnp\" (UniqueName: \"kubernetes.io/projected/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-kube-api-access-52xnp\") pod \"maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq\" (UID: \"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"
Oct 30 13:04:36.621446 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.591582    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq\" (UID: \"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"
Oct 30 13:04:36.621446 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.591641    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq\" (UID: \"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"
Oct 30 13:04:36.696712 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.695225    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq\" (UID: \"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"
Oct 30 13:04:36.696712 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.695488    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52xnp\" (UniqueName: \"kubernetes.io/projected/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-kube-api-access-52xnp\") pod \"maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq\" (UID: \"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"
Oct 30 13:04:36.696712 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.695576    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq\" (UID: \"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"
Oct 30 13:04:36.696712 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.696380    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq\" (UID: \"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"
Oct 30 13:04:36.709176 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.709145    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq\" (UID: \"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"
Oct 30 13:04:36.737686 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:36.737651    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52xnp\" (UniqueName: \"kubernetes.io/projected/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-kube-api-access-52xnp\") pod \"maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq\" (UID: \"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"
Oct 30 13:04:37.007852 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:37.007817    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"
Oct 30 13:04:37.119430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:37.117926    5837 generic.go:356] "Generic (PLEG): container finished" podID="9f22bb1d-f0c7-436e-b0d2-810e9098e80b" containerID="bc133416c9a9331b05457c01458d0ed8bc908e876db3cfb55f96e3d2b4b66631" exitCode=0
Oct 30 13:04:37.119430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:37.118084    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg" event={"ID":"9f22bb1d-f0c7-436e-b0d2-810e9098e80b","Type":"ContainerDied","Data":"bc133416c9a9331b05457c01458d0ed8bc908e876db3cfb55f96e3d2b4b66631"}
Oct 30 13:04:37.559526 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:37.558986    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:37.559526 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:37.559068    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:37.700160 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:37.699996    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" path="/var/lib/kubelet/pods/9e2ab740-6bfa-438b-a878-e8a852a9b1a1/volumes"
Oct 30 13:04:38.422081 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:38.421973    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:38.845974 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:38.840735    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:04:39.096487 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:39.081878    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"]
Oct 30 13:04:39.437492 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:39.437446    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:39.443610 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:39.443566    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:04:39.500581 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:39.456113    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 13:04:39.813420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:39.773520    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": context deadline exceeded"
Oct 30 13:04:40.269477 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:40.221117    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:40.457532 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:40.457476    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:40.494479 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:40.459214    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded"
Oct 30 13:04:40.573862 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:40.506495    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:40.578420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:40.573446    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:40.578688 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:40.578653    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:40.650674 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:40.650646    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 13:04:40.652039 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:40.652013    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq" event={"ID":"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c","Type":"ContainerStarted","Data":"cf12dadcd34d6d786edba202c093607602b833a60ac25f87d162fc0fdc44aff8"}
Oct 30 13:04:40.652988 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:40.652968    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:04:40.852641 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:40.852499    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:40.852641 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:40.852569    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" podUID="faf87439-5044-43cd-9b1a-200188299f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:41.246486 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:41.237753    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:41.272437 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:41.260509    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:41.272437 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:41.260579    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:41.303165 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:41.303114    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:41.303547 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:41.303487    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" podUID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:41.495430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:41.495221    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 13:04:41.614481 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:41.602525    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:42.207352 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:42.207274    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"
Oct 30 13:04:42.238529 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:42.238481    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:42.363922 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:42.363876    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-maas-api-openshift-ca\") pod \"9f22bb1d-f0c7-436e-b0d2-810e9098e80b\" (UID: \"9f22bb1d-f0c7-436e-b0d2-810e9098e80b\") "
Oct 30 13:04:42.364498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:42.364467    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-kube-api-access-bh9tb\" (UniqueName: \"kubernetes.io/projected/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-kube-api-access-bh9tb\") pod \"9f22bb1d-f0c7-436e-b0d2-810e9098e80b\" (UID: \"9f22bb1d-f0c7-436e-b0d2-810e9098e80b\") "
Oct 30 13:04:42.364782 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:42.364761    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-maas-api-xks-ca\") pod \"9f22bb1d-f0c7-436e-b0d2-810e9098e80b\" (UID: \"9f22bb1d-f0c7-436e-b0d2-810e9098e80b\") "
Oct 30 13:04:42.379978 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:42.379942    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-maas-api-openshift-ca" pod "9f22bb1d-f0c7-436e-b0d2-810e9098e80b" (UID: "9f22bb1d-f0c7-436e-b0d2-810e9098e80b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:42.501320 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:42.475929    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:42.501320 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:42.500811    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:42.571433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:42.562558    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-kube-api-access-bh9tb" pod "9f22bb1d-f0c7-436e-b0d2-810e9098e80b" (UID: "9f22bb1d-f0c7-436e-b0d2-810e9098e80b"). InnerVolumeSpecName "kube-api-access-bh9tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:42.571433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:42.562926    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-maas-api-xks-ca" pod "9f22bb1d-f0c7-436e-b0d2-810e9098e80b" (UID: "9f22bb1d-f0c7-436e-b0d2-810e9098e80b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:42.616607 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:42.585791    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh9tb\" (UniqueName: \"kubernetes.io/projected/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-kube-api-access-bh9tb\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:42.616607 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:42.585829    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f22bb1d-f0c7-436e-b0d2-810e9098e80b-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:42.616607 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:42.616177    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:43.242521 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:43.239543    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:43.351469 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:43.348939    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg" event={"ID":"9f22bb1d-f0c7-436e-b0d2-810e9098e80b","Type":"ContainerDied","Data":"16e4de8ac37be6657e5aa33c4aef2afd3fc15badc5580fd33b7b3be29fa2a224"}
Oct 30 13:04:43.351469 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:43.349051    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16e4de8ac37be6657e5aa33c4aef2afd3fc15badc5580fd33b7b3be29fa2a224"
Oct 30 13:04:43.351469 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:43.349158    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"
Oct 30 13:04:43.433421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:43.430908    5837 generic.go:356] "Generic (PLEG): container finished" podID="4ad6701d-ebdb-4ad6-88ab-92c70a35c41c" containerID="6a48b9a09db21bb004cfea570a57563173e3a03512df3910f2d31e3ad54a8da3" exitCode=0
Oct 30 13:04:43.433421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:43.431043    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq" event={"ID":"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c","Type":"ContainerDied","Data":"6a48b9a09db21bb004cfea570a57563173e3a03512df3910f2d31e3ad54a8da3"}
Oct 30 13:04:43.517796 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:43.516490    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 13:04:43.620540 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:43.620491    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:44.266419 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:44.240966    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:44.454417 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:44.451233    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": context deadline exceeded"
Oct 30 13:04:44.533641 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:44.533529    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:44.776422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:44.640365    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:45.344420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:45.328772    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:45.566422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:45.565752    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:45.761609 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:45.761560    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:46.079160 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:04:46.079067    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:04:46.337323 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:46.334518    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:04:46.428618 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:46.428591    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 13:04:46.538437 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:46.519880    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 13:04:46.649512 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:46.609134    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:04:46.767423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:46.767325    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"
Oct 30 13:04:46.849848 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:46.848122    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-maas-api-openshift-ca\") pod \"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c\" (UID: \"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c\") "
Oct 30 13:04:46.849848 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:46.848201    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-kube-api-access-52xnp\" (UniqueName: \"kubernetes.io/projected/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-kube-api-access-52xnp\") pod \"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c\" (UID: \"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c\") "
Oct 30 13:04:46.849848 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:46.848379    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-maas-api-xks-ca\") pod \"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c\" (UID: \"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c\") "
Oct 30 13:04:46.850512 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:46.850483    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-maas-api-openshift-ca" pod "4ad6701d-ebdb-4ad6-88ab-92c70a35c41c" (UID: "4ad6701d-ebdb-4ad6-88ab-92c70a35c41c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:46.864676 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:46.864639    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-kube-api-access-52xnp" pod "4ad6701d-ebdb-4ad6-88ab-92c70a35c41c" (UID: "4ad6701d-ebdb-4ad6-88ab-92c70a35c41c"). InnerVolumeSpecName "kube-api-access-52xnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:46.868574 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:46.868527    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-maas-api-xks-ca" pod "4ad6701d-ebdb-4ad6-88ab-92c70a35c41c" (UID: "4ad6701d-ebdb-4ad6-88ab-92c70a35c41c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:46.963408 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:46.962701    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:46.963408 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:46.962734    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:46.963408 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:46.962779    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52xnp\" (UniqueName: \"kubernetes.io/projected/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c-kube-api-access-52xnp\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:47.157944 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:47.157907    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq" event={"ID":"4ad6701d-ebdb-4ad6-88ab-92c70a35c41c","Type":"ContainerDied","Data":"cf12dadcd34d6d786edba202c093607602b833a60ac25f87d162fc0fdc44aff8"}
Oct 30 13:04:47.157944 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:47.157929    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"
Oct 30 13:04:47.158187 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:47.157962    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf12dadcd34d6d786edba202c093607602b833a60ac25f87d162fc0fdc44aff8"
Oct 30 13:04:50.129013 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:50.128943    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f_d630a8a0-5896-415e-8c51-34bcd9486545/payload-processing/0.log"
Oct 30 13:04:50.297032 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:50.297007    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f_d630a8a0-5896-415e-8c51-34bcd9486545/payload-processing/0.log"
Oct 30 13:04:52.809312 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.809066    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:52.841837 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.809872    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:52.841837 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.822614    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"]
Oct 30 13:04:52.841837 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.823344    5837 cpu_manager.go:414] "containerMap: removing container" podUID="4ad6701d-ebdb-4ad6-88ab-92c70a35c41c" containerName="revoke-keys"
Oct 30 13:04:52.841837 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.823355    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad6701d-ebdb-4ad6-88ab-92c70a35c41c" containerName="revoke-keys"
Oct 30 13:04:52.841837 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.823377    5837 cpu_manager.go:414] "containerMap: removing container" podUID="9f22bb1d-f0c7-436e-b0d2-810e9098e80b" containerName="revoke-keys"
Oct 30 13:04:52.841837 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.823383    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f22bb1d-f0c7-436e-b0d2-810e9098e80b" containerName="revoke-keys"
Oct 30 13:04:52.841837 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.823503    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad6701d-ebdb-4ad6-88ab-92c70a35c41c" containerName="revoke-keys"
Oct 30 13:04:52.841837 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.823515    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e2ab740-6bfa-438b-a878-e8a852a9b1a1" containerName="istio-proxy"
Oct 30 13:04:52.841837 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.823523    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f22bb1d-f0c7-436e-b0d2-810e9098e80b" containerName="revoke-keys"
Oct 30 13:04:52.882149 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.882116    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"]
Oct 30 13:04:52.882308 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.882247    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"
Oct 30 13:04:52.897023 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.896939    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:04:52.908577 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.908548    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:04:52.908577 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.908551    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:04:52.908770 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.908551    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:04:52.992072 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.992036    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7kcd\" (UniqueName: \"kubernetes.io/projected/2728869d-2a7c-4530-8f53-93309ebf4230-kube-api-access-x7kcd\") pod \"maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh\" (UID: \"2728869d-2a7c-4530-8f53-93309ebf4230\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"
Oct 30 13:04:52.992277 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.992258    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2728869d-2a7c-4530-8f53-93309ebf4230-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh\" (UID: \"2728869d-2a7c-4530-8f53-93309ebf4230\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"
Oct 30 13:04:52.992380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:52.992366    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2728869d-2a7c-4530-8f53-93309ebf4230-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh\" (UID: \"2728869d-2a7c-4530-8f53-93309ebf4230\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"
Oct 30 13:04:53.094276 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.094184    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7kcd\" (UniqueName: \"kubernetes.io/projected/2728869d-2a7c-4530-8f53-93309ebf4230-kube-api-access-x7kcd\") pod \"maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh\" (UID: \"2728869d-2a7c-4530-8f53-93309ebf4230\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"
Oct 30 13:04:53.094860 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.094326    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2728869d-2a7c-4530-8f53-93309ebf4230-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh\" (UID: \"2728869d-2a7c-4530-8f53-93309ebf4230\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"
Oct 30 13:04:53.094860 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.094381    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2728869d-2a7c-4530-8f53-93309ebf4230-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh\" (UID: \"2728869d-2a7c-4530-8f53-93309ebf4230\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"
Oct 30 13:04:53.095494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.095463    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2728869d-2a7c-4530-8f53-93309ebf4230-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh\" (UID: \"2728869d-2a7c-4530-8f53-93309ebf4230\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"
Oct 30 13:04:53.097488 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.096503    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": read tcp 10.134.0.2:37424->10.134.0.192:15021: read: connection reset by peer" start-of-body=
Oct 30 13:04:53.097488 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.096552    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": read tcp 10.134.0.2:37424->10.134.0.192:15021: read: connection reset by peer"
Oct 30 13:04:53.097716 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.097584    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2728869d-2a7c-4530-8f53-93309ebf4230-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh\" (UID: \"2728869d-2a7c-4530-8f53-93309ebf4230\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"
Oct 30 13:04:53.113952 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.113925    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7kcd\" (UniqueName: \"kubernetes.io/projected/2728869d-2a7c-4530-8f53-93309ebf4230-kube-api-access-x7kcd\") pod \"maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh\" (UID: \"2728869d-2a7c-4530-8f53-93309ebf4230\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"
Oct 30 13:04:53.221288 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.221253    5837 generic.go:356] "Generic (PLEG): container finished" podID="612f5032-bf59-40f9-b043-e20395d385b2" containerID="1f685f1dee63842b48ade8a9418369c0122cade121a5ab5604ef91e408638800" exitCode=0
Oct 30 13:04:53.221288 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.221285    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" event={"ID":"612f5032-bf59-40f9-b043-e20395d385b2","Type":"ContainerDied","Data":"1f685f1dee63842b48ade8a9418369c0122cade121a5ab5604ef91e408638800"}
Oct 30 13:04:53.221924 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.221872    5837 scope.go:122] "RemoveContainer" containerID="1f685f1dee63842b48ade8a9418369c0122cade121a5ab5604ef91e408638800"
Oct 30 13:04:53.224873 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.224831    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:53.225127 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.225097    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:53.225288 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.225164    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:53.245899 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.245872    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"
Oct 30 13:04:53.646736 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:53.646690    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"]
Oct 30 13:04:53.648432 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:04:53.648406    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2728869d_2a7c_4530_8f53_93309ebf4230.slice/crio-abe67a70f0fc7251a082df4e80099eba1442901860156818a72024302b099c41 WatchSource:0}: Error finding container abe67a70f0fc7251a082df4e80099eba1442901860156818a72024302b099c41: Status 404 returned error can't find the container with id abe67a70f0fc7251a082df4e80099eba1442901860156818a72024302b099c41
Oct 30 13:04:54.232277 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:54.232246    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" event={"ID":"612f5032-bf59-40f9-b043-e20395d385b2","Type":"ContainerStarted","Data":"4f5bed7edfad0ee5957c1eb14b931952d5ca8db0c6cf7d61dcd9a782ce510ced"}
Oct 30 13:04:54.234307 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:54.234280    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh" event={"ID":"2728869d-2a7c-4530-8f53-93309ebf4230","Type":"ContainerStarted","Data":"1767a306031a69eada49e1b2d4bad043f292d904c2c948602bc8c41a93b2f892"}
Oct 30 13:04:54.234520 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:54.234326    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh" event={"ID":"2728869d-2a7c-4530-8f53-93309ebf4230","Type":"ContainerStarted","Data":"abe67a70f0fc7251a082df4e80099eba1442901860156818a72024302b099c41"}
Oct 30 13:04:54.327284 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:54.327146    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh" podStartSLOduration=2.327132288 podStartE2EDuration="2.327132288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:54.325597501 +0000 UTC m=+1682.849881740" watchObservedRunningTime="2026-10-30 13:04:54.327132288 +0000 UTC m=+1682.851416507"
Oct 30 13:04:54.403016 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:54.402975    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:04:54.403196 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:54.403130    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 13:04:54.403196 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:54.403177    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 13:04:55.246430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:55.246329    5837 generic.go:356] "Generic (PLEG): container finished" podID="2728869d-2a7c-4530-8f53-93309ebf4230" containerID="1767a306031a69eada49e1b2d4bad043f292d904c2c948602bc8c41a93b2f892" exitCode=0
Oct 30 13:04:55.247105 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:55.246447    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh" event={"ID":"2728869d-2a7c-4530-8f53-93309ebf4230","Type":"ContainerDied","Data":"1767a306031a69eada49e1b2d4bad043f292d904c2c948602bc8c41a93b2f892"}
Oct 30 13:04:55.405425 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:55.404924    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 13:04:55.405651 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:55.405020    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 13:04:56.257723 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.257692    5837 generic.go:356] "Generic (PLEG): container finished" podID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerID="3f11e4b460fb43ff4546577e286bbb4010d22453f87ba60ea169e58a6e0f7f48" exitCode=0
Oct 30 13:04:56.258274 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.257770    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" event={"ID":"f163e2e9-ace0-44b6-962c-19b61fe31c95","Type":"ContainerDied","Data":"3f11e4b460fb43ff4546577e286bbb4010d22453f87ba60ea169e58a6e0f7f48"}
Oct 30 13:04:56.258728 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.258442    5837 scope.go:122] "RemoveContainer" containerID="3f11e4b460fb43ff4546577e286bbb4010d22453f87ba60ea169e58a6e0f7f48"
Oct 30 13:04:56.260117 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.260087    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:56.260212 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.260158    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:56.260212 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.260190    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:56.413596 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.413547    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 13:04:56.413800 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.413634    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 13:04:56.593867 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.591663    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"
Oct 30 13:04:56.702424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.696837    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2728869d-2a7c-4530-8f53-93309ebf4230-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2728869d-2a7c-4530-8f53-93309ebf4230-maas-api-openshift-ca\") pod \"2728869d-2a7c-4530-8f53-93309ebf4230\" (UID: \"2728869d-2a7c-4530-8f53-93309ebf4230\") "
Oct 30 13:04:56.702424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.697155    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2728869d-2a7c-4530-8f53-93309ebf4230-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2728869d-2a7c-4530-8f53-93309ebf4230-maas-api-xks-ca\") pod \"2728869d-2a7c-4530-8f53-93309ebf4230\" (UID: \"2728869d-2a7c-4530-8f53-93309ebf4230\") "
Oct 30 13:04:56.702424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.697294    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2728869d-2a7c-4530-8f53-93309ebf4230-kube-api-access-x7kcd\" (UniqueName: \"kubernetes.io/projected/2728869d-2a7c-4530-8f53-93309ebf4230-kube-api-access-x7kcd\") pod \"2728869d-2a7c-4530-8f53-93309ebf4230\" (UID: \"2728869d-2a7c-4530-8f53-93309ebf4230\") "
Oct 30 13:04:56.715733 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.715139    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2728869d-2a7c-4530-8f53-93309ebf4230-maas-api-openshift-ca" pod "2728869d-2a7c-4530-8f53-93309ebf4230" (UID: "2728869d-2a7c-4530-8f53-93309ebf4230"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:56.723420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.719203    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2728869d-2a7c-4530-8f53-93309ebf4230-maas-api-xks-ca" pod "2728869d-2a7c-4530-8f53-93309ebf4230" (UID: "2728869d-2a7c-4530-8f53-93309ebf4230"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:56.751969 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.747482    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"]
Oct 30 13:04:56.751969 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.748860    5837 cpu_manager.go:414] "containerMap: removing container" podUID="2728869d-2a7c-4530-8f53-93309ebf4230" containerName="revoke-keys"
Oct 30 13:04:56.751969 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.748885    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2728869d-2a7c-4530-8f53-93309ebf4230" containerName="revoke-keys"
Oct 30 13:04:56.751969 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.749070    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2728869d-2a7c-4530-8f53-93309ebf4230" containerName="revoke-keys"
Oct 30 13:04:56.838459 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.837203    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2728869d-2a7c-4530-8f53-93309ebf4230-kube-api-access-x7kcd" pod "2728869d-2a7c-4530-8f53-93309ebf4230" (UID: "2728869d-2a7c-4530-8f53-93309ebf4230"). InnerVolumeSpecName "kube-api-access-x7kcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:56.847824 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.847683    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7kcd\" (UniqueName: \"kubernetes.io/projected/2728869d-2a7c-4530-8f53-93309ebf4230-kube-api-access-x7kcd\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:56.847824 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.847718    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2728869d-2a7c-4530-8f53-93309ebf4230-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:56.847824 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.847732    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2728869d-2a7c-4530-8f53-93309ebf4230-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:04:56.850520 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.848945    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:56.851063 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.848742    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"]
Oct 30 13:04:56.867090 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.867064    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-666577-openshift-default-dockercfg-tvs2z\""
Oct 30 13:04:56.961649 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.961602    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-credential-socket\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:56.961998 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.961976    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istiod-ca-cert\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:56.962159 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.962143    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-data\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:56.962301 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.962286    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nhsb\" (UniqueName: \"kubernetes.io/projected/12c0a91a-fa80-4a17-9278-6bc43b6b9847-kube-api-access-5nhsb\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:56.969590 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.969549    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-podinfo\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:56.969943 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.969925    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-token\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:56.970088 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.970074    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-workload-certs\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:56.977638 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.977609    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-workload-socket\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:56.977865 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:56.977850    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-envoy\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.037469 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.037343    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"]
Oct 30 13:04:57.052933 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.052886    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"]
Oct 30 13:04:57.080225 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.080170    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-credential-socket\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.080679 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.080645    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istiod-ca-cert\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.080832 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.080718    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-data\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.080832 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.080754    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nhsb\" (UniqueName: \"kubernetes.io/projected/12c0a91a-fa80-4a17-9278-6bc43b6b9847-kube-api-access-5nhsb\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.080832 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.080816    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-podinfo\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.081033 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.080917    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-token\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.081033 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.080942    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-workload-certs\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.081159 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.081039    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-workload-socket\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.081159 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.081062    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-envoy\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.081272 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.081220    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istiod-ca-cert\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.081344 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.080649    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-credential-socket\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.081581 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.081558    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-workload-certs\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.082049 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.081957    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-workload-socket\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.082207 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.082164    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-data\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.132132 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.103806    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-envoy\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.132132 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.123562    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-podinfo\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.162171 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.162145    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-token\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.173221 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.173175    5837 scope.go:122] "RemoveContainer" containerID="b232c1a77a174d8fc9e20a9feef337908deaff8708d553433c019e376285e2da"
Oct 30 13:04:57.232682 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.228589    5837 scope.go:122] "RemoveContainer" containerID="ee2545b750b4b722df43e3a22a2977f94e138c2faa735cba3ce16fbe09d7a5b4"
Oct 30 13:04:57.242961 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.242933    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nhsb\" (UniqueName: \"kubernetes.io/projected/12c0a91a-fa80-4a17-9278-6bc43b6b9847-kube-api-access-5nhsb\") pod \"e2e-praxis-b-666577-openshift-default-f887989df-gvxxs\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") " pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.277106 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.275749    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"
Oct 30 13:04:57.277106 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.276682    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh" event={"ID":"2728869d-2a7c-4530-8f53-93309ebf4230","Type":"ContainerDied","Data":"abe67a70f0fc7251a082df4e80099eba1442901860156818a72024302b099c41"}
Oct 30 13:04:57.277106 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.276774    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abe67a70f0fc7251a082df4e80099eba1442901860156818a72024302b099c41"
Oct 30 13:04:57.411326 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.407709    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 13:04:57.411326 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.407789    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 13:04:57.428425 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.414870    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" event={"ID":"f163e2e9-ace0-44b6-962c-19b61fe31c95","Type":"ContainerStarted","Data":"06a664ce2615e78d052b6912976f70f30db9f34c5b2cbf3c006260482514bec4"}
Oct 30 13:04:57.509428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.509275    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:57.696951 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.696813    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:04:57.697242 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.697220    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:04:57.697445 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:57.697418    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 13:04:58.310419 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:58.283430    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"]
Oct 30 13:04:58.345478 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:58.345434    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:58.345652 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:58.345516    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:58.345652 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:58.345544    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:58.419424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:58.417751    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 13:04:58.419424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:58.417809    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 13:04:58.455763 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:58.455737    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs" event={"ID":"12c0a91a-fa80-4a17-9278-6bc43b6b9847","Type":"ContainerStarted","Data":"cad5c1820f85e5552c84fd2212ee1a081ce39758476ec6b6a3162390e550788a"}
Oct 30 13:04:58.716479 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:58.698884    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:04:58.716479 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:58.698956    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 13:04:59.422423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:59.419951    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 13:04:59.422423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:59.420023    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 13:04:59.472662 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:59.472630    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs" event={"ID":"12c0a91a-fa80-4a17-9278-6bc43b6b9847","Type":"ContainerStarted","Data":"90ed49c8e55f0938be4d90747c666e3f6cdd7c4f59c568ab77f7e87bf7efd192"}
Oct 30 13:04:59.536319 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:59.536103    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs" podStartSLOduration=3.536082989 podStartE2EDuration="3.536082989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:59.533938434 +0000 UTC m=+1688.058222679" watchObservedRunningTime="2026-10-30 13:04:59.536082989 +0000 UTC m=+1688.060367209"
Oct 30 13:04:59.544516 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:59.544476    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:59.550242 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:59.550221    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:04:59.698017 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:59.697927    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:04:59.698290 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:04:59.698252    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 13:05:00.409911 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:00.409871    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 13:05:00.410184 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:00.410151    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 13:05:00.481468 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:00.481434    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:05:00.482664 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:00.482644    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:05:00.694926 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:00.694826    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:05:00.695204 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:00.695173    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 13:05:00.981200 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:00.981174    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"]
Oct 30 13:05:01.017451 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.010635    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.023994 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.021922    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"]
Oct 30 13:05:01.027876 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.027193    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-4cb0fe\"/\"kube-root-ca.crt\""
Oct 30 13:05:01.027876 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.027303    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-4cb0fe\"/\"openshift-service-ca.crt\""
Oct 30 13:05:01.027876 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.027373    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-4cb0fe\"/\"default-dockercfg-4msmb\""
Oct 30 13:05:01.027876 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.027513    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-4cb0fe\"/\"praxis-route-4cb0fe-kserve-self-signed-certs\""
Oct 30 13:05:01.069097 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.069064    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlzr7\" (UniqueName: \"kubernetes.io/projected/bb128524-fe08-4334-8ffe-86220d1e62b8-kube-api-access-jlzr7\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.069273 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.069178    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-home\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.069273 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.069239    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-model-cache\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.069417 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.069273    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-dshm\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.069541 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.069521    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb128524-fe08-4334-8ffe-86220d1e62b8-tls-certs\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.069579 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.069562    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-tmp-dir\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.170750 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.170709    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb128524-fe08-4334-8ffe-86220d1e62b8-tls-certs\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.170957 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.170771    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-tmp-dir\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.170957 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.170866    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlzr7\" (UniqueName: \"kubernetes.io/projected/bb128524-fe08-4334-8ffe-86220d1e62b8-kube-api-access-jlzr7\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.170957 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.170932    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-home\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.171135 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.170994    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-model-cache\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.171135 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.171046    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-dshm\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.171247 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.171218    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-tmp-dir\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.171309 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.171253    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-home\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.171440 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.171386    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-model-cache\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.173457 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.173389    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-dshm\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.173578 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.173565    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb128524-fe08-4334-8ffe-86220d1e62b8-tls-certs\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.225604 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.219551    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlzr7\" (UniqueName: \"kubernetes.io/projected/bb128524-fe08-4334-8ffe-86220d1e62b8-kube-api-access-jlzr7\") pod \"praxis-route-4cb0fe-kserve-754974d7b5-zbp54\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") " pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.339476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.338737    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:01.430420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.423836    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:05:01.430420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.425556    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:05:01.452447 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.435786    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:05:01.718642 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.718549    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:05:01.719312 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.719275    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 13:05:01.811577 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:01.811544    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"]
Oct 30 13:05:01.812113 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:05:01.812090    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb128524_fe08_4334_8ffe_86220d1e62b8.slice/crio-babcdf498564b59e679e90cdcf0db44e651e1f9eae5f76f2c039ac1122f44ea6 WatchSource:0}: Error finding container babcdf498564b59e679e90cdcf0db44e651e1f9eae5f76f2c039ac1122f44ea6: Status 404 returned error can't find the container with id babcdf498564b59e679e90cdcf0db44e651e1f9eae5f76f2c039ac1122f44ea6
Oct 30 13:05:02.038364 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.038319    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r" podUID="158886bc-5fe6-4182-b481-389fc298abd8" containerName="payload-processing" containerID="cri-o://6b272fa59925e8567a19005dfc0dabaf9d388f24e5a36f5f31041dc0e3ae4325" gracePeriod=25
Oct 30 13:05:02.054067 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.054033    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt" podUID="79da32ac-98d8-481a-9713-7e784f1c2ce7" containerName="payload-pre-processing" containerID="cri-o://1af96a80cf6cc27d3925c4d3d79f86f21e6c32a1c83328427519765ccd64bb5a" gracePeriod=25
Oct 30 13:05:02.376899 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.376873    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"
Oct 30 13:05:02.438929 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.438900    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158886bc-5fe6-4182-b481-389fc298abd8-kube-api-access-c768w\" (UniqueName: \"kubernetes.io/projected/158886bc-5fe6-4182-b481-389fc298abd8-kube-api-access-c768w\") pod \"158886bc-5fe6-4182-b481-389fc298abd8\" (UID: \"158886bc-5fe6-4182-b481-389fc298abd8\") "
Oct 30 13:05:02.439067 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.438977    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158886bc-5fe6-4182-b481-389fc298abd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158886bc-5fe6-4182-b481-389fc298abd8-plugins-config-volume\") pod \"158886bc-5fe6-4182-b481-389fc298abd8\" (UID: \"158886bc-5fe6-4182-b481-389fc298abd8\") "
Oct 30 13:05:02.439338 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.439317    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158886bc-5fe6-4182-b481-389fc298abd8-plugins-config-volume" pod "158886bc-5fe6-4182-b481-389fc298abd8" (UID: "158886bc-5fe6-4182-b481-389fc298abd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:02.439724 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.439702    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158886bc-5fe6-4182-b481-389fc298abd8-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:02.441282 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.441257    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158886bc-5fe6-4182-b481-389fc298abd8-kube-api-access-c768w" pod "158886bc-5fe6-4182-b481-389fc298abd8" (UID: "158886bc-5fe6-4182-b481-389fc298abd8"). InnerVolumeSpecName "kube-api-access-c768w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:02.456911 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.456891    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"
Oct 30 13:05:02.540817 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.540787    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79da32ac-98d8-481a-9713-7e784f1c2ce7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79da32ac-98d8-481a-9713-7e784f1c2ce7-plugins-config-volume\") pod \"79da32ac-98d8-481a-9713-7e784f1c2ce7\" (UID: \"79da32ac-98d8-481a-9713-7e784f1c2ce7\") "
Oct 30 13:05:02.541094 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.541071    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79da32ac-98d8-481a-9713-7e784f1c2ce7-kube-api-access-thflw\" (UniqueName: \"kubernetes.io/projected/79da32ac-98d8-481a-9713-7e784f1c2ce7-kube-api-access-thflw\") pod \"79da32ac-98d8-481a-9713-7e784f1c2ce7\" (UID: \"79da32ac-98d8-481a-9713-7e784f1c2ce7\") "
Oct 30 13:05:02.541209 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.541188    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79da32ac-98d8-481a-9713-7e784f1c2ce7-plugins-config-volume" pod "79da32ac-98d8-481a-9713-7e784f1c2ce7" (UID: "79da32ac-98d8-481a-9713-7e784f1c2ce7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:02.541773 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.541753    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79da32ac-98d8-481a-9713-7e784f1c2ce7-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:02.541773 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.541772    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c768w\" (UniqueName: \"kubernetes.io/projected/158886bc-5fe6-4182-b481-389fc298abd8-kube-api-access-c768w\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:02.543670 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.543645    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79da32ac-98d8-481a-9713-7e784f1c2ce7-kube-api-access-thflw" pod "79da32ac-98d8-481a-9713-7e784f1c2ce7" (UID: "79da32ac-98d8-481a-9713-7e784f1c2ce7"). InnerVolumeSpecName "kube-api-access-thflw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:02.547381 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.547354    5837 generic.go:356] "Generic (PLEG): container finished" podID="79da32ac-98d8-481a-9713-7e784f1c2ce7" containerID="1af96a80cf6cc27d3925c4d3d79f86f21e6c32a1c83328427519765ccd64bb5a" exitCode=0
Oct 30 13:05:02.547511 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.547419    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"
Oct 30 13:05:02.547511 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.547429    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt" event={"ID":"79da32ac-98d8-481a-9713-7e784f1c2ce7","Type":"ContainerDied","Data":"1af96a80cf6cc27d3925c4d3d79f86f21e6c32a1c83328427519765ccd64bb5a"}
Oct 30 13:05:02.547652 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.547533    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt" event={"ID":"79da32ac-98d8-481a-9713-7e784f1c2ce7","Type":"ContainerDied","Data":"7fe431ad68ed7f16a2e4c21ce05e5c6c8e08e0ee30cf00508cac243d76b967ae"}
Oct 30 13:05:02.547652 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.547571    5837 scope.go:122] "RemoveContainer" containerID="1af96a80cf6cc27d3925c4d3d79f86f21e6c32a1c83328427519765ccd64bb5a"
Oct 30 13:05:02.549531 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.549506    5837 generic.go:356] "Generic (PLEG): container finished" podID="158886bc-5fe6-4182-b481-389fc298abd8" containerID="6b272fa59925e8567a19005dfc0dabaf9d388f24e5a36f5f31041dc0e3ae4325" exitCode=0
Oct 30 13:05:02.549641 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.549555    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"
Oct 30 13:05:02.549641 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.549618    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r" event={"ID":"158886bc-5fe6-4182-b481-389fc298abd8","Type":"ContainerDied","Data":"6b272fa59925e8567a19005dfc0dabaf9d388f24e5a36f5f31041dc0e3ae4325"}
Oct 30 13:05:02.549767 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.549658    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r" event={"ID":"158886bc-5fe6-4182-b481-389fc298abd8","Type":"ContainerDied","Data":"5386381bdb420b828942935527a26e5f80f07e66c915a8493100cdec0bd09385"}
Oct 30 13:05:02.551929 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.551906    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" event={"ID":"bb128524-fe08-4334-8ffe-86220d1e62b8","Type":"ContainerStarted","Data":"e42d9cad13590bb8d372bffac3c60c7f6827a1d1908cee1561dbc672c0291449"}
Oct 30 13:05:02.552042 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.551943    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" event={"ID":"bb128524-fe08-4334-8ffe-86220d1e62b8","Type":"ContainerStarted","Data":"babcdf498564b59e679e90cdcf0db44e651e1f9eae5f76f2c039ac1122f44ea6"}
Oct 30 13:05:02.560616 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.560597    5837 scope.go:122] "RemoveContainer" containerID="1af96a80cf6cc27d3925c4d3d79f86f21e6c32a1c83328427519765ccd64bb5a"
Oct 30 13:05:02.561126 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:05:02.560992    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af96a80cf6cc27d3925c4d3d79f86f21e6c32a1c83328427519765ccd64bb5a\": container with ID starting with 1af96a80cf6cc27d3925c4d3d79f86f21e6c32a1c83328427519765ccd64bb5a not found: ID does not exist" containerID="1af96a80cf6cc27d3925c4d3d79f86f21e6c32a1c83328427519765ccd64bb5a"
Oct 30 13:05:02.561126 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.561018    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af96a80cf6cc27d3925c4d3d79f86f21e6c32a1c83328427519765ccd64bb5a"} err="failed to get container status \"1af96a80cf6cc27d3925c4d3d79f86f21e6c32a1c83328427519765ccd64bb5a\": rpc error: code = NotFound desc = could not find container \"1af96a80cf6cc27d3925c4d3d79f86f21e6c32a1c83328427519765ccd64bb5a\": container with ID starting with 1af96a80cf6cc27d3925c4d3d79f86f21e6c32a1c83328427519765ccd64bb5a not found: ID does not exist"
Oct 30 13:05:02.561126 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.561048    5837 scope.go:122] "RemoveContainer" containerID="6b272fa59925e8567a19005dfc0dabaf9d388f24e5a36f5f31041dc0e3ae4325"
Oct 30 13:05:02.572930 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.572904    5837 scope.go:122] "RemoveContainer" containerID="6b272fa59925e8567a19005dfc0dabaf9d388f24e5a36f5f31041dc0e3ae4325"
Oct 30 13:05:02.573274 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:05:02.573253    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b272fa59925e8567a19005dfc0dabaf9d388f24e5a36f5f31041dc0e3ae4325\": container with ID starting with 6b272fa59925e8567a19005dfc0dabaf9d388f24e5a36f5f31041dc0e3ae4325 not found: ID does not exist" containerID="6b272fa59925e8567a19005dfc0dabaf9d388f24e5a36f5f31041dc0e3ae4325"
Oct 30 13:05:02.573362 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.573293    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b272fa59925e8567a19005dfc0dabaf9d388f24e5a36f5f31041dc0e3ae4325"} err="failed to get container status \"6b272fa59925e8567a19005dfc0dabaf9d388f24e5a36f5f31041dc0e3ae4325\": rpc error: code = NotFound desc = could not find container \"6b272fa59925e8567a19005dfc0dabaf9d388f24e5a36f5f31041dc0e3ae4325\": container with ID starting with 6b272fa59925e8567a19005dfc0dabaf9d388f24e5a36f5f31041dc0e3ae4325 not found: ID does not exist"
Oct 30 13:05:02.644135 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.644111    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thflw\" (UniqueName: \"kubernetes.io/projected/79da32ac-98d8-481a-9713-7e784f1c2ce7-kube-api-access-thflw\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:02.671828 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.671785    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podStartSLOduration=2.6717714470000002 podStartE2EDuration="2.671771447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:02.669796845 +0000 UTC m=+1691.194081065" watchObservedRunningTime="2026-10-30 13:05:02.671771447 +0000 UTC m=+1691.196055683"
Oct 30 13:05:02.708973 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.708930    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:05:02.709148 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.708983    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 13:05:02.730244 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.730213    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"]
Oct 30 13:05:02.737238 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.737214    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ba9a8c-554db57548-lp65r"]
Oct 30 13:05:02.894731 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.889322    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"]
Oct 30 13:05:02.940629 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:02.940599    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ba9a8c-57d4f5cb47-7v5bt"]
Oct 30 13:05:03.340022 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:03.339983    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:03.428546 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:03.428512    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"]
Oct 30 13:05:03.619071 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:03.619041    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158886bc-5fe6-4182-b481-389fc298abd8" path="/var/lib/kubelet/pods/158886bc-5fe6-4182-b481-389fc298abd8/volumes"
Oct 30 13:05:03.619666 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:03.619647    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79da32ac-98d8-481a-9713-7e784f1c2ce7" path="/var/lib/kubelet/pods/79da32ac-98d8-481a-9713-7e784f1c2ce7/volumes"
Oct 30 13:05:03.694710 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:03.694670    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:05:03.694933 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:03.694735    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 13:05:04.700980 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.700945    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:05:04.701770 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.701746    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:05:04.702640 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.702621    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:05:04.883592 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.883557    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"]
Oct 30 13:05:04.884244 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.884230    5837 cpu_manager.go:414] "containerMap: removing container" podUID="158886bc-5fe6-4182-b481-389fc298abd8" containerName="payload-processing"
Oct 30 13:05:04.884332 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.884245    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158886bc-5fe6-4182-b481-389fc298abd8" containerName="payload-processing"
Oct 30 13:05:04.884332 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.884291    5837 cpu_manager.go:414] "containerMap: removing container" podUID="79da32ac-98d8-481a-9713-7e784f1c2ce7" containerName="payload-pre-processing"
Oct 30 13:05:04.884332 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.884297    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79da32ac-98d8-481a-9713-7e784f1c2ce7" containerName="payload-pre-processing"
Oct 30 13:05:04.884549 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.884424    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="79da32ac-98d8-481a-9713-7e784f1c2ce7" containerName="payload-pre-processing"
Oct 30 13:05:04.884549 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.884437    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="158886bc-5fe6-4182-b481-389fc298abd8" containerName="payload-processing"
Oct 30 13:05:04.888366 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.888340    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"
Oct 30 13:05:04.906058 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.906037    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-666577\""
Oct 30 13:05:04.907594 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.907577    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"]
Oct 30 13:05:04.960612 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.960543    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"]
Oct 30 13:05:04.968741 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.968706    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"
Oct 30 13:05:04.983977 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.983950    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-666577-dockercfg-8v745\""
Oct 30 13:05:04.984830 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:04.984809    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"]
Oct 30 13:05:05.001208 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.001174    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db24l\" (UniqueName: \"kubernetes.io/projected/663b46aa-7286-46ae-b760-d56bf2cf7d5d-kube-api-access-db24l\") pod \"payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9\" (UID: \"663b46aa-7286-46ae-b760-d56bf2cf7d5d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"
Oct 30 13:05:05.005540 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.001290    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9fbc5c-80d9-4674-b16e-dd48110d1fad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6\" (UID: \"8a9fbc5c-80d9-4674-b16e-dd48110d1fad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"
Oct 30 13:05:05.005540 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.001383    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663b46aa-7286-46ae-b760-d56bf2cf7d5d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9\" (UID: \"663b46aa-7286-46ae-b760-d56bf2cf7d5d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"
Oct 30 13:05:05.005540 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.001528    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mc9k\" (UniqueName: \"kubernetes.io/projected/8a9fbc5c-80d9-4674-b16e-dd48110d1fad-kube-api-access-5mc9k\") pod \"payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6\" (UID: \"8a9fbc5c-80d9-4674-b16e-dd48110d1fad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"
Oct 30 13:05:05.138436 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.112763    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mc9k\" (UniqueName: \"kubernetes.io/projected/8a9fbc5c-80d9-4674-b16e-dd48110d1fad-kube-api-access-5mc9k\") pod \"payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6\" (UID: \"8a9fbc5c-80d9-4674-b16e-dd48110d1fad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"
Oct 30 13:05:05.138935 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.138907    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db24l\" (UniqueName: \"kubernetes.io/projected/663b46aa-7286-46ae-b760-d56bf2cf7d5d-kube-api-access-db24l\") pod \"payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9\" (UID: \"663b46aa-7286-46ae-b760-d56bf2cf7d5d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"
Oct 30 13:05:05.148054 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.143584    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9fbc5c-80d9-4674-b16e-dd48110d1fad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6\" (UID: \"8a9fbc5c-80d9-4674-b16e-dd48110d1fad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"
Oct 30 13:05:05.176420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.150623    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663b46aa-7286-46ae-b760-d56bf2cf7d5d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9\" (UID: \"663b46aa-7286-46ae-b760-d56bf2cf7d5d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"
Oct 30 13:05:05.177010 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.158350    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9fbc5c-80d9-4674-b16e-dd48110d1fad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6\" (UID: \"8a9fbc5c-80d9-4674-b16e-dd48110d1fad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"
Oct 30 13:05:05.211609 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.205547    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663b46aa-7286-46ae-b760-d56bf2cf7d5d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9\" (UID: \"663b46aa-7286-46ae-b760-d56bf2cf7d5d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"
Oct 30 13:05:05.255966 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.255931    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mc9k\" (UniqueName: \"kubernetes.io/projected/8a9fbc5c-80d9-4674-b16e-dd48110d1fad-kube-api-access-5mc9k\") pod \"payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6\" (UID: \"8a9fbc5c-80d9-4674-b16e-dd48110d1fad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"
Oct 30 13:05:05.264946 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.264909    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db24l\" (UniqueName: \"kubernetes.io/projected/663b46aa-7286-46ae-b760-d56bf2cf7d5d-kube-api-access-db24l\") pod \"payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9\" (UID: \"663b46aa-7286-46ae-b760-d56bf2cf7d5d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"
Oct 30 13:05:05.286214 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.286185    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"
Oct 30 13:05:05.501521 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.501490    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"
Oct 30 13:05:05.595439 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.595357    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"]
Oct 30 13:05:05.603962 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:05:05.603924    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9fbc5c_80d9_4674_b16e_dd48110d1fad.slice/crio-b688cf1fc9b20b4763d01b4b9bb22b3d130638cec4bbc04d255d37bba43a8f74 WatchSource:0}: Error finding container b688cf1fc9b20b4763d01b4b9bb22b3d130638cec4bbc04d255d37bba43a8f74: Status 404 returned error can't find the container with id b688cf1fc9b20b4763d01b4b9bb22b3d130638cec4bbc04d255d37bba43a8f74
Oct 30 13:05:05.807074 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:05:05.807044    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663b46aa_7286_46ae_b760_d56bf2cf7d5d.slice/crio-0dd7388c924e473ba45fdfac3a1e84655ddec88d36c8683a9217a536bc6d8c1a WatchSource:0}: Error finding container 0dd7388c924e473ba45fdfac3a1e84655ddec88d36c8683a9217a536bc6d8c1a: Status 404 returned error can't find the container with id 0dd7388c924e473ba45fdfac3a1e84655ddec88d36c8683a9217a536bc6d8c1a
Oct 30 13:05:05.807074 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:05.807053    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"]
Oct 30 13:05:06.593147 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:06.593115    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6" event={"ID":"8a9fbc5c-80d9-4674-b16e-dd48110d1fad","Type":"ContainerStarted","Data":"6c7aaed2b66c812e0fc44e7291dca93c6c4ab3e7a8f56eb87378e57068c9dfdd"}
Oct 30 13:05:06.593347 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:06.593166    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6" event={"ID":"8a9fbc5c-80d9-4674-b16e-dd48110d1fad","Type":"ContainerStarted","Data":"b688cf1fc9b20b4763d01b4b9bb22b3d130638cec4bbc04d255d37bba43a8f74"}
Oct 30 13:05:06.593347 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:06.593309    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"
Oct 30 13:05:06.595093 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:06.595042    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9" event={"ID":"663b46aa-7286-46ae-b760-d56bf2cf7d5d","Type":"ContainerStarted","Data":"f548c1a9f2a74a54fc6b4f0654f24c40207cb77131970f7f85378f7819834572"}
Oct 30 13:05:06.595093 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:06.595087    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9" event={"ID":"663b46aa-7286-46ae-b760-d56bf2cf7d5d","Type":"ContainerStarted","Data":"0dd7388c924e473ba45fdfac3a1e84655ddec88d36c8683a9217a536bc6d8c1a"}
Oct 30 13:05:06.635386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:06.635340    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6" podStartSLOduration=2.6353270970000002 podStartE2EDuration="2.635327097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:06.631712107 +0000 UTC m=+1695.155996326" watchObservedRunningTime="2026-10-30 13:05:06.635327097 +0000 UTC m=+1695.159611315"
Oct 30 13:05:06.671893 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:06.671842    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9" podStartSLOduration=2.671828398 podStartE2EDuration="2.671828398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:06.669810627 +0000 UTC m=+1695.194094858" watchObservedRunningTime="2026-10-30 13:05:06.671828398 +0000 UTC m=+1695.196112617"
Oct 30 13:05:07.606908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:07.606880    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"
Oct 30 13:05:08.428940 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:08.428899    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk" podUID="56eb33e2-cc60-4372-8b87-e369b9c12140" containerName="payload-pre-processing" containerID="cri-o://f275cd314954c7ce0d23fe70c38ad6bd00164a00b340349eebd7a3ee40ad2d1a" gracePeriod=25
Oct 30 13:05:08.788545 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:08.788506    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk" podUID="56eb33e2-cc60-4372-8b87-e369b9c12140" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:05:11.254552 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.254520    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"]
Oct 30 13:05:11.258521 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.258500    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.277351 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.276968    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"]
Oct 30 13:05:11.279342 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.279313    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tsqw\" (UniqueName: \"kubernetes.io/projected/403d9d4d-2f47-4d38-96de-32158a75a865-kube-api-access-8tsqw\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.279543 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.279514    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-model-cache\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.279644 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.279624    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-dshm\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.279723 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.279705    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-tmp-dir\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.279830 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.279816    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/403d9d4d-2f47-4d38-96de-32158a75a865-tls-certs\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.279900 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.279884    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-home\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.282729 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.282706    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-026c2f\"/\"openshift-service-ca.crt\""
Oct 30 13:05:11.282729 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.282712    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-026c2f\"/\"kube-root-ca.crt\""
Oct 30 13:05:11.287512 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.287494    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-026c2f\"/\"default-dockercfg-m6bsd\""
Oct 30 13:05:11.287602 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.287515    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-026c2f\"/\"ipp-route-026c2f-kserve-self-signed-certs\""
Oct 30 13:05:11.340106 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.340067    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:11.352732 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.352708    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:11.380963 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.380930    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-tmp-dir\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.381137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.381039    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/403d9d4d-2f47-4d38-96de-32158a75a865-tls-certs\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.381137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.381085    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-home\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.381225 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.381159    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tsqw\" (UniqueName: \"kubernetes.io/projected/403d9d4d-2f47-4d38-96de-32158a75a865-kube-api-access-8tsqw\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.381282 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.381241    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-model-cache\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.381282 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.381269    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-dshm\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.381543 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.381512    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-home\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.381660 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.381643    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-tmp-dir\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.381660 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.381650    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-model-cache\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.383615 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.383591    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-dshm\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.383817 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.383799    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/403d9d4d-2f47-4d38-96de-32158a75a865-tls-certs\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.402620 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.402594    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tsqw\" (UniqueName: \"kubernetes.io/projected/403d9d4d-2f47-4d38-96de-32158a75a865-kube-api-access-8tsqw\") pod \"ipp-route-026c2f-kserve-b495b7649-ljhhd\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") " pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.423153 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.423120    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"]
Oct 30 13:05:11.427071 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.427050    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"
Oct 30 13:05:11.440912 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.440881    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"]
Oct 30 13:05:11.442630 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.442607    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:05:11.445312 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.445296    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"
Oct 30 13:05:11.445603 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.445581    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:05:11.445692 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.445637    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:05:11.445754 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.445701    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:05:11.445807 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.445794    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"]
Oct 30 13:05:11.465342 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.465294    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"]
Oct 30 13:05:11.483932 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.483171    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7824bafb-5wc5n\" (UID: \"b2c59a19-6c67-42f0-9703-7dbf1d4b6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"
Oct 30 13:05:11.483932 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.483232    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7fd7\" (UniqueName: \"kubernetes.io/projected/587f599f-eac3-4046-926f-9ee4a4f391e4-kube-api-access-f7fd7\") pod \"maas-api-revoke-sub-3598d2e5-n22nk\" (UID: \"587f599f-eac3-4046-926f-9ee4a4f391e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"
Oct 30 13:05:11.483932 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.483277    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktmp2\" (UniqueName: \"kubernetes.io/projected/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-kube-api-access-ktmp2\") pod \"maas-api-revoke-sub-7824bafb-5wc5n\" (UID: \"b2c59a19-6c67-42f0-9703-7dbf1d4b6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"
Oct 30 13:05:11.483932 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.483347    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7824bafb-5wc5n\" (UID: \"b2c59a19-6c67-42f0-9703-7dbf1d4b6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"
Oct 30 13:05:11.483932 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.483552    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587f599f-eac3-4046-926f-9ee4a4f391e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3598d2e5-n22nk\" (UID: \"587f599f-eac3-4046-926f-9ee4a4f391e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"
Oct 30 13:05:11.483932 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.483651    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587f599f-eac3-4046-926f-9ee4a4f391e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3598d2e5-n22nk\" (UID: \"587f599f-eac3-4046-926f-9ee4a4f391e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"
Oct 30 13:05:11.580422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.580061    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:11.586860 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.586826    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587f599f-eac3-4046-926f-9ee4a4f391e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3598d2e5-n22nk\" (UID: \"587f599f-eac3-4046-926f-9ee4a4f391e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"
Oct 30 13:05:11.587122 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.587103    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587f599f-eac3-4046-926f-9ee4a4f391e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3598d2e5-n22nk\" (UID: \"587f599f-eac3-4046-926f-9ee4a4f391e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"
Oct 30 13:05:11.587361 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.587337    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7824bafb-5wc5n\" (UID: \"b2c59a19-6c67-42f0-9703-7dbf1d4b6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"
Oct 30 13:05:11.588692 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.588671    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7fd7\" (UniqueName: \"kubernetes.io/projected/587f599f-eac3-4046-926f-9ee4a4f391e4-kube-api-access-f7fd7\") pod \"maas-api-revoke-sub-3598d2e5-n22nk\" (UID: \"587f599f-eac3-4046-926f-9ee4a4f391e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"
Oct 30 13:05:11.588852 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.588836    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmp2\" (UniqueName: \"kubernetes.io/projected/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-kube-api-access-ktmp2\") pod \"maas-api-revoke-sub-7824bafb-5wc5n\" (UID: \"b2c59a19-6c67-42f0-9703-7dbf1d4b6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"
Oct 30 13:05:11.589092 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.589076    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7824bafb-5wc5n\" (UID: \"b2c59a19-6c67-42f0-9703-7dbf1d4b6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"
Oct 30 13:05:11.592796 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.592766    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7824bafb-5wc5n\" (UID: \"b2c59a19-6c67-42f0-9703-7dbf1d4b6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"
Oct 30 13:05:11.594998 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.593427    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587f599f-eac3-4046-926f-9ee4a4f391e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3598d2e5-n22nk\" (UID: \"587f599f-eac3-4046-926f-9ee4a4f391e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"
Oct 30 13:05:11.674580 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.674248    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7824bafb-5wc5n\" (UID: \"b2c59a19-6c67-42f0-9703-7dbf1d4b6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"
Oct 30 13:05:11.676201 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.676158    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587f599f-eac3-4046-926f-9ee4a4f391e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3598d2e5-n22nk\" (UID: \"587f599f-eac3-4046-926f-9ee4a4f391e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"
Oct 30 13:05:11.682759 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.679321    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktmp2\" (UniqueName: \"kubernetes.io/projected/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-kube-api-access-ktmp2\") pod \"maas-api-revoke-sub-7824bafb-5wc5n\" (UID: \"b2c59a19-6c67-42f0-9703-7dbf1d4b6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"
Oct 30 13:05:11.682759 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.679535    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7fd7\" (UniqueName: \"kubernetes.io/projected/587f599f-eac3-4046-926f-9ee4a4f391e4-kube-api-access-f7fd7\") pod \"maas-api-revoke-sub-3598d2e5-n22nk\" (UID: \"587f599f-eac3-4046-926f-9ee4a4f391e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"
Oct 30 13:05:11.740450 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.739881    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:11.744076 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.744047    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"
Oct 30 13:05:11.764045 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.764020    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"
Oct 30 13:05:11.945835 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:11.945804    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"]
Oct 30 13:05:11.947447 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:05:11.947416    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod403d9d4d_2f47_4d38_96de_32158a75a865.slice/crio-dcc6e371922349336679aa879b1fd5b110d472b5983826246c1079c6fc07e39d WatchSource:0}: Error finding container dcc6e371922349336679aa879b1fd5b110d472b5983826246c1079c6fc07e39d: Status 404 returned error can't find the container with id dcc6e371922349336679aa879b1fd5b110d472b5983826246c1079c6fc07e39d
Oct 30 13:05:12.116874 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:05:12.116844    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587f599f_eac3_4046_926f_9ee4a4f391e4.slice/crio-acc4d55765d3ce8c2abf142a6c19121b28633704d7ccd7ea025effd9d9abed08 WatchSource:0}: Error finding container acc4d55765d3ce8c2abf142a6c19121b28633704d7ccd7ea025effd9d9abed08: Status 404 returned error can't find the container with id acc4d55765d3ce8c2abf142a6c19121b28633704d7ccd7ea025effd9d9abed08
Oct 30 13:05:12.118007 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:12.117978    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"]
Oct 30 13:05:12.131183 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:12.131154    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"]
Oct 30 13:05:12.133601 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:05:12.133573    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c59a19_6c67_42f0_9703_7dbf1d4b6489.slice/crio-f8461ea2319e759bf475e4bfa0ab007834483d65dff613d83ebbda1662d35b87 WatchSource:0}: Error finding container f8461ea2319e759bf475e4bfa0ab007834483d65dff613d83ebbda1662d35b87: Status 404 returned error can't find the container with id f8461ea2319e759bf475e4bfa0ab007834483d65dff613d83ebbda1662d35b87
Oct 30 13:05:12.720194 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:12.720109    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk" event={"ID":"587f599f-eac3-4046-926f-9ee4a4f391e4","Type":"ContainerStarted","Data":"ac91f51da3e98933b443880c5348fb40b141b7a6bb67eba499462c570b42701b"}
Oct 30 13:05:12.720194 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:12.720167    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk" event={"ID":"587f599f-eac3-4046-926f-9ee4a4f391e4","Type":"ContainerStarted","Data":"acc4d55765d3ce8c2abf142a6c19121b28633704d7ccd7ea025effd9d9abed08"}
Oct 30 13:05:12.721903 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:12.721875    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" event={"ID":"403d9d4d-2f47-4d38-96de-32158a75a865","Type":"ContainerStarted","Data":"555cff26e076f709f3a0104378a2d1113a493acd129059308eabb3990087764b"}
Oct 30 13:05:12.722056 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:12.721915    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" event={"ID":"403d9d4d-2f47-4d38-96de-32158a75a865","Type":"ContainerStarted","Data":"dcc6e371922349336679aa879b1fd5b110d472b5983826246c1079c6fc07e39d"}
Oct 30 13:05:12.723823 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:12.723797    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n" event={"ID":"b2c59a19-6c67-42f0-9703-7dbf1d4b6489","Type":"ContainerStarted","Data":"be5914392ddfc64ea6e846efec26808f42fc3cac69b8aedb9df33875c0e12946"}
Oct 30 13:05:12.723930 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:12.723837    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n" event={"ID":"b2c59a19-6c67-42f0-9703-7dbf1d4b6489","Type":"ContainerStarted","Data":"f8461ea2319e759bf475e4bfa0ab007834483d65dff613d83ebbda1662d35b87"}
Oct 30 13:05:12.757974 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:12.757921    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk" podStartSLOduration=1.757905818 podStartE2EDuration="1.757905818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:12.757130988 +0000 UTC m=+1701.281415210" watchObservedRunningTime="2026-10-30 13:05:12.757905818 +0000 UTC m=+1701.282190038"
Oct 30 13:05:12.797560 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:12.797510    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podStartSLOduration=1.7974946649999999 podStartE2EDuration="1.797494665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:12.794644093 +0000 UTC m=+1701.318928312" watchObservedRunningTime="2026-10-30 13:05:12.797494665 +0000 UTC m=+1701.321778885"
Oct 30 13:05:12.834196 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:12.834148    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n" podStartSLOduration=1.834134466 podStartE2EDuration="1.834134466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:12.832925257 +0000 UTC m=+1701.357209477" watchObservedRunningTime="2026-10-30 13:05:12.834134466 +0000 UTC m=+1701.358418718"
Oct 30 13:05:13.580946 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:13.580914    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:13.732963 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:13.732930    5837 generic.go:356] "Generic (PLEG): container finished" podID="b2c59a19-6c67-42f0-9703-7dbf1d4b6489" containerID="be5914392ddfc64ea6e846efec26808f42fc3cac69b8aedb9df33875c0e12946" exitCode=0
Oct 30 13:05:13.733463 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:13.732981    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n" event={"ID":"b2c59a19-6c67-42f0-9703-7dbf1d4b6489","Type":"ContainerDied","Data":"be5914392ddfc64ea6e846efec26808f42fc3cac69b8aedb9df33875c0e12946"}
Oct 30 13:05:13.734798 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:13.734776    5837 generic.go:356] "Generic (PLEG): container finished" podID="587f599f-eac3-4046-926f-9ee4a4f391e4" containerID="ac91f51da3e98933b443880c5348fb40b141b7a6bb67eba499462c570b42701b" exitCode=0
Oct 30 13:05:13.734925 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:13.734834    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk" event={"ID":"587f599f-eac3-4046-926f-9ee4a4f391e4","Type":"ContainerDied","Data":"ac91f51da3e98933b443880c5348fb40b141b7a6bb67eba499462c570b42701b"}
Oct 30 13:05:14.877875 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:14.877843    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"]
Oct 30 13:05:14.886751 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:14.886727    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ba9a8c-fsgjq"]
Oct 30 13:05:14.999347 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:14.991010    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"
Oct 30 13:05:15.060904 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.060878    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"
Oct 30 13:05:15.100154 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:05:15.100114    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:05:15.103170 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.103138    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-maas-api-xks-ca\") pod \"b2c59a19-6c67-42f0-9703-7dbf1d4b6489\" (UID: \"b2c59a19-6c67-42f0-9703-7dbf1d4b6489\") "
Oct 30 13:05:15.103352 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.103333    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-kube-api-access-ktmp2\" (UniqueName: \"kubernetes.io/projected/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-kube-api-access-ktmp2\") pod \"b2c59a19-6c67-42f0-9703-7dbf1d4b6489\" (UID: \"b2c59a19-6c67-42f0-9703-7dbf1d4b6489\") "
Oct 30 13:05:15.103452 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.103421    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-maas-api-openshift-ca\") pod \"b2c59a19-6c67-42f0-9703-7dbf1d4b6489\" (UID: \"b2c59a19-6c67-42f0-9703-7dbf1d4b6489\") "
Oct 30 13:05:15.104009 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.103981    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-maas-api-openshift-ca" pod "b2c59a19-6c67-42f0-9703-7dbf1d4b6489" (UID: "b2c59a19-6c67-42f0-9703-7dbf1d4b6489"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:15.104172 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.104151    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:15.106090 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.106062    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-maas-api-xks-ca" pod "b2c59a19-6c67-42f0-9703-7dbf1d4b6489" (UID: "b2c59a19-6c67-42f0-9703-7dbf1d4b6489"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:15.106546 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.106525    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-kube-api-access-ktmp2" pod "b2c59a19-6c67-42f0-9703-7dbf1d4b6489" (UID: "b2c59a19-6c67-42f0-9703-7dbf1d4b6489"). InnerVolumeSpecName "kube-api-access-ktmp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:15.205611 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.205572    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587f599f-eac3-4046-926f-9ee4a4f391e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587f599f-eac3-4046-926f-9ee4a4f391e4-maas-api-openshift-ca\") pod \"587f599f-eac3-4046-926f-9ee4a4f391e4\" (UID: \"587f599f-eac3-4046-926f-9ee4a4f391e4\") "
Oct 30 13:05:15.205788 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.205704    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/587f599f-eac3-4046-926f-9ee4a4f391e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587f599f-eac3-4046-926f-9ee4a4f391e4-maas-api-xks-ca\") pod \"587f599f-eac3-4046-926f-9ee4a4f391e4\" (UID: \"587f599f-eac3-4046-926f-9ee4a4f391e4\") "
Oct 30 13:05:15.205788 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.205783    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587f599f-eac3-4046-926f-9ee4a4f391e4-kube-api-access-f7fd7\" (UniqueName: \"kubernetes.io/projected/587f599f-eac3-4046-926f-9ee4a4f391e4-kube-api-access-f7fd7\") pod \"587f599f-eac3-4046-926f-9ee4a4f391e4\" (UID: \"587f599f-eac3-4046-926f-9ee4a4f391e4\") "
Oct 30 13:05:15.205911 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.205884    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587f599f-eac3-4046-926f-9ee4a4f391e4-maas-api-openshift-ca" pod "587f599f-eac3-4046-926f-9ee4a4f391e4" (UID: "587f599f-eac3-4046-926f-9ee4a4f391e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:15.206276 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.206256    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:15.206350 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.206281    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktmp2\" (UniqueName: \"kubernetes.io/projected/b2c59a19-6c67-42f0-9703-7dbf1d4b6489-kube-api-access-ktmp2\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:15.206350 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.206294    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587f599f-eac3-4046-926f-9ee4a4f391e4-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:15.207893 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.207875    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587f599f-eac3-4046-926f-9ee4a4f391e4-maas-api-xks-ca" pod "587f599f-eac3-4046-926f-9ee4a4f391e4" (UID: "587f599f-eac3-4046-926f-9ee4a4f391e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:15.207984 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.207964    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587f599f-eac3-4046-926f-9ee4a4f391e4-kube-api-access-f7fd7" pod "587f599f-eac3-4046-926f-9ee4a4f391e4" (UID: "587f599f-eac3-4046-926f-9ee4a4f391e4"). InnerVolumeSpecName "kube-api-access-f7fd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:15.308210 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.308146    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587f599f-eac3-4046-926f-9ee4a4f391e4-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:15.308210 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.308167    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7fd7\" (UniqueName: \"kubernetes.io/projected/587f599f-eac3-4046-926f-9ee4a4f391e4-kube-api-access-f7fd7\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:15.607156 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.607078    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad6701d-ebdb-4ad6-88ab-92c70a35c41c" path="/var/lib/kubelet/pods/4ad6701d-ebdb-4ad6-88ab-92c70a35c41c/volumes"
Oct 30 13:05:15.745069 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.745040    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"]
Oct 30 13:05:15.745247 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.745210    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw" podUID="8fa68fa1-c682-4b28-ad60-7c3207842d6a" containerName="istio-proxy" containerID="cri-o://ef1f24ddd2b406b0be258da8ebe2d4c0c91d3610d154e1fd00fc56f523238496" gracePeriod=30
Oct 30 13:05:15.758817 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.758789    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"
Oct 30 13:05:15.758817 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.758801    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n" event={"ID":"b2c59a19-6c67-42f0-9703-7dbf1d4b6489","Type":"ContainerDied","Data":"f8461ea2319e759bf475e4bfa0ab007834483d65dff613d83ebbda1662d35b87"}
Oct 30 13:05:15.759032 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.758861    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8461ea2319e759bf475e4bfa0ab007834483d65dff613d83ebbda1662d35b87"
Oct 30 13:05:15.761706 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.761676    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk" event={"ID":"587f599f-eac3-4046-926f-9ee4a4f391e4","Type":"ContainerDied","Data":"acc4d55765d3ce8c2abf142a6c19121b28633704d7ccd7ea025effd9d9abed08"}
Oct 30 13:05:15.761867 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.761728    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acc4d55765d3ce8c2abf142a6c19121b28633704d7ccd7ea025effd9d9abed08"
Oct 30 13:05:15.761867 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:15.761789    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"
Oct 30 13:05:17.607487 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:17.607457    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"
Oct 30 13:05:18.629332 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:18.629299    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"
Oct 30 13:05:18.826662 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:18.826610    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk" podUID="56eb33e2-cc60-4372-8b87-e369b9c12140" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:05:21.035947 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.035923    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 13:05:21.103844 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.103067    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-credential-socket\") pod \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") "
Oct 30 13:05:21.104056 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.103909    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-podinfo\") pod \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") "
Oct 30 13:05:21.104056 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.103935    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-workload-certs\") pod \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") "
Oct 30 13:05:21.104056 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.103988    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-envoy\") pod \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") "
Oct 30 13:05:21.104276 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.104098    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-data\") pod \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") "
Oct 30 13:05:21.104276 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.104142    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-token\" (UniqueName: \"kubernetes.io/projected/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-token\") pod \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") "
Oct 30 13:05:21.104276 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.104167    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-credential-socket" pod "8fa68fa1-c682-4b28-ad60-7c3207842d6a" (UID: "8fa68fa1-c682-4b28-ad60-7c3207842d6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:21.104276 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.104180    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa68fa1-c682-4b28-ad60-7c3207842d6a-kube-api-access-bx989\" (UniqueName: \"kubernetes.io/projected/8fa68fa1-c682-4b28-ad60-7c3207842d6a-kube-api-access-bx989\") pod \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") "
Oct 30 13:05:21.104276 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.104254    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istiod-ca-cert\") pod \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") "
Oct 30 13:05:21.104594 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.104416    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-workload-socket\") pod \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\" (UID: \"8fa68fa1-c682-4b28-ad60-7c3207842d6a\") "
Oct 30 13:05:21.104649 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.104580    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-workload-certs" pod "8fa68fa1-c682-4b28-ad60-7c3207842d6a" (UID: "8fa68fa1-c682-4b28-ad60-7c3207842d6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:21.104890 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.104863    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-workload-socket" pod "8fa68fa1-c682-4b28-ad60-7c3207842d6a" (UID: "8fa68fa1-c682-4b28-ad60-7c3207842d6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:21.104890 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.104862    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istiod-ca-cert" pod "8fa68fa1-c682-4b28-ad60-7c3207842d6a" (UID: "8fa68fa1-c682-4b28-ad60-7c3207842d6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:21.105115 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.105096    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-data" pod "8fa68fa1-c682-4b28-ad60-7c3207842d6a" (UID: "8fa68fa1-c682-4b28-ad60-7c3207842d6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:21.105338 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.105319    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:21.106259 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.106242    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:21.106382 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.106361    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:21.107875 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.106382    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:21.107971 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.107898    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:21.107971 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.107529    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-podinfo" pod "8fa68fa1-c682-4b28-ad60-7c3207842d6a" (UID: "8fa68fa1-c682-4b28-ad60-7c3207842d6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:05:21.107971 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.107540    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-token" pod "8fa68fa1-c682-4b28-ad60-7c3207842d6a" (UID: "8fa68fa1-c682-4b28-ad60-7c3207842d6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:21.107971 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.107670    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa68fa1-c682-4b28-ad60-7c3207842d6a-kube-api-access-bx989" pod "8fa68fa1-c682-4b28-ad60-7c3207842d6a" (UID: "8fa68fa1-c682-4b28-ad60-7c3207842d6a"). InnerVolumeSpecName "kube-api-access-bx989". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:21.107971 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.107781    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-envoy" pod "8fa68fa1-c682-4b28-ad60-7c3207842d6a" (UID: "8fa68fa1-c682-4b28-ad60-7c3207842d6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:21.209329 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.209292    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:21.209329 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.209322    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx989\" (UniqueName: \"kubernetes.io/projected/8fa68fa1-c682-4b28-ad60-7c3207842d6a-kube-api-access-bx989\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:21.209329 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.209331    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:21.209592 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.209340    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa68fa1-c682-4b28-ad60-7c3207842d6a-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:21.581473 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.581430    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:21.595117 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.595087    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:21.868764 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.868681    5837 generic.go:356] "Generic (PLEG): container finished" podID="8fa68fa1-c682-4b28-ad60-7c3207842d6a" containerID="ef1f24ddd2b406b0be258da8ebe2d4c0c91d3610d154e1fd00fc56f523238496" exitCode=0
Oct 30 13:05:21.868764 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.868729    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"
Oct 30 13:05:21.868956 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.868756    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw" event={"ID":"8fa68fa1-c682-4b28-ad60-7c3207842d6a","Type":"ContainerDied","Data":"ef1f24ddd2b406b0be258da8ebe2d4c0c91d3610d154e1fd00fc56f523238496"}
Oct 30 13:05:21.868956 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.868811    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw" event={"ID":"8fa68fa1-c682-4b28-ad60-7c3207842d6a","Type":"ContainerDied","Data":"1d40db9e7d695e17a3bee7e6ffb2b611d4cf75a9a42496b412b2585adf90e5c2"}
Oct 30 13:05:21.868956 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.868834    5837 scope.go:122] "RemoveContainer" containerID="ef1f24ddd2b406b0be258da8ebe2d4c0c91d3610d154e1fd00fc56f523238496"
Oct 30 13:05:21.880824 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.880806    5837 scope.go:122] "RemoveContainer" containerID="ef1f24ddd2b406b0be258da8ebe2d4c0c91d3610d154e1fd00fc56f523238496"
Oct 30 13:05:21.881070 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.881047    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:21.881165 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:05:21.881094    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef1f24ddd2b406b0be258da8ebe2d4c0c91d3610d154e1fd00fc56f523238496\": container with ID starting with ef1f24ddd2b406b0be258da8ebe2d4c0c91d3610d154e1fd00fc56f523238496 not found: ID does not exist" containerID="ef1f24ddd2b406b0be258da8ebe2d4c0c91d3610d154e1fd00fc56f523238496"
Oct 30 13:05:21.881165 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:21.881120    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1f24ddd2b406b0be258da8ebe2d4c0c91d3610d154e1fd00fc56f523238496"} err="failed to get container status \"ef1f24ddd2b406b0be258da8ebe2d4c0c91d3610d154e1fd00fc56f523238496\": rpc error: code = NotFound desc = could not find container \"ef1f24ddd2b406b0be258da8ebe2d4c0c91d3610d154e1fd00fc56f523238496\": container with ID starting with ef1f24ddd2b406b0be258da8ebe2d4c0c91d3610d154e1fd00fc56f523238496 not found: ID does not exist"
Oct 30 13:05:22.003655 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.003614    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"]
Oct 30 13:05:22.017784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.017748    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-v7qbw"]
Oct 30 13:05:22.031805 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.031763    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"]
Oct 30 13:05:22.032151 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.032121    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw" podUID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerName="istio-proxy" containerID="cri-o://1f9605a12a898d1943bd473ac3272ae1ca6ccb723afdacee30e6e0a0b36de7a9" gracePeriod=30
Oct 30 13:05:22.559448 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.559412    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"]
Oct 30 13:05:22.560238 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.560222    5837 cpu_manager.go:414] "containerMap: removing container" podUID="8fa68fa1-c682-4b28-ad60-7c3207842d6a" containerName="istio-proxy"
Oct 30 13:05:22.560289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.560241    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa68fa1-c682-4b28-ad60-7c3207842d6a" containerName="istio-proxy"
Oct 30 13:05:22.560289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.560270    5837 cpu_manager.go:414] "containerMap: removing container" podUID="b2c59a19-6c67-42f0-9703-7dbf1d4b6489" containerName="revoke-keys"
Oct 30 13:05:22.560289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.560275    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c59a19-6c67-42f0-9703-7dbf1d4b6489" containerName="revoke-keys"
Oct 30 13:05:22.560405 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.560305    5837 cpu_manager.go:414] "containerMap: removing container" podUID="587f599f-eac3-4046-926f-9ee4a4f391e4" containerName="revoke-keys"
Oct 30 13:05:22.560405 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.560311    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587f599f-eac3-4046-926f-9ee4a4f391e4" containerName="revoke-keys"
Oct 30 13:05:22.560496 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.560428    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="587f599f-eac3-4046-926f-9ee4a4f391e4" containerName="revoke-keys"
Oct 30 13:05:22.560496 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.560439    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa68fa1-c682-4b28-ad60-7c3207842d6a" containerName="istio-proxy"
Oct 30 13:05:22.560496 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.560448    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c59a19-6c67-42f0-9703-7dbf1d4b6489" containerName="revoke-keys"
Oct 30 13:05:22.564536 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.564511    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"
Oct 30 13:05:22.590476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.586709    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:05:22.590476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.586715    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:05:22.590476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.586760    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:05:22.590476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.586715    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:05:22.593383 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.593342    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"]
Oct 30 13:05:22.652677 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.652645    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b017696f-0424-4f3b-b1bb-20c6d0997076-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2\" (UID: \"b017696f-0424-4f3b-b1bb-20c6d0997076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"
Oct 30 13:05:22.652677 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.652679    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2kfm\" (UniqueName: \"kubernetes.io/projected/b017696f-0424-4f3b-b1bb-20c6d0997076-kube-api-access-v2kfm\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2\" (UID: \"b017696f-0424-4f3b-b1bb-20c6d0997076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"
Oct 30 13:05:22.653016 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.652986    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b017696f-0424-4f3b-b1bb-20c6d0997076-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2\" (UID: \"b017696f-0424-4f3b-b1bb-20c6d0997076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"
Oct 30 13:05:22.754285 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.754252    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b017696f-0424-4f3b-b1bb-20c6d0997076-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2\" (UID: \"b017696f-0424-4f3b-b1bb-20c6d0997076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"
Oct 30 13:05:22.754451 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.754354    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b017696f-0424-4f3b-b1bb-20c6d0997076-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2\" (UID: \"b017696f-0424-4f3b-b1bb-20c6d0997076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"
Oct 30 13:05:22.754451 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.754373    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2kfm\" (UniqueName: \"kubernetes.io/projected/b017696f-0424-4f3b-b1bb-20c6d0997076-kube-api-access-v2kfm\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2\" (UID: \"b017696f-0424-4f3b-b1bb-20c6d0997076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"
Oct 30 13:05:22.754902 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.754882    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b017696f-0424-4f3b-b1bb-20c6d0997076-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2\" (UID: \"b017696f-0424-4f3b-b1bb-20c6d0997076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"
Oct 30 13:05:22.756998 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.756976    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b017696f-0424-4f3b-b1bb-20c6d0997076-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2\" (UID: \"b017696f-0424-4f3b-b1bb-20c6d0997076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"
Oct 30 13:05:22.773526 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.773504    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2kfm\" (UniqueName: \"kubernetes.io/projected/b017696f-0424-4f3b-b1bb-20c6d0997076-kube-api-access-v2kfm\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2\" (UID: \"b017696f-0424-4f3b-b1bb-20c6d0997076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"
Oct 30 13:05:22.889174 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:22.889096    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"
Oct 30 13:05:23.195442 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:05:23.195408    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb017696f_0424_4f3b_b1bb_20c6d0997076.slice/crio-e99280d4e3904fa4a93270e7ebce253b50401ae05795139d849672dc384d75cf WatchSource:0}: Error finding container e99280d4e3904fa4a93270e7ebce253b50401ae05795139d849672dc384d75cf: Status 404 returned error can't find the container with id e99280d4e3904fa4a93270e7ebce253b50401ae05795139d849672dc384d75cf
Oct 30 13:05:23.195571 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:23.195479    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"]
Oct 30 13:05:23.606491 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:23.606464    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa68fa1-c682-4b28-ad60-7c3207842d6a" path="/var/lib/kubelet/pods/8fa68fa1-c682-4b28-ad60-7c3207842d6a/volumes"
Oct 30 13:05:23.890749 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:23.890658    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2" event={"ID":"b017696f-0424-4f3b-b1bb-20c6d0997076","Type":"ContainerStarted","Data":"0c3d7494654c47c20648ffe7a03e6be04cf7760c947e5b342d22d644bea9706f"}
Oct 30 13:05:23.890749 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:23.890706    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2" event={"ID":"b017696f-0424-4f3b-b1bb-20c6d0997076","Type":"ContainerStarted","Data":"e99280d4e3904fa4a93270e7ebce253b50401ae05795139d849672dc384d75cf"}
Oct 30 13:05:23.937205 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:23.937159    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2" podStartSLOduration=1.9371447769999999 podStartE2EDuration="1.937144777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:23.933616814 +0000 UTC m=+1712.457901219" watchObservedRunningTime="2026-10-30 13:05:23.937144777 +0000 UTC m=+1712.461429014"
Oct 30 13:05:24.900158 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:24.900125    5837 generic.go:356] "Generic (PLEG): container finished" podID="b017696f-0424-4f3b-b1bb-20c6d0997076" containerID="0c3d7494654c47c20648ffe7a03e6be04cf7760c947e5b342d22d644bea9706f" exitCode=0
Oct 30 13:05:24.900555 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:24.900213    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2" event={"ID":"b017696f-0424-4f3b-b1bb-20c6d0997076","Type":"ContainerDied","Data":"0c3d7494654c47c20648ffe7a03e6be04cf7760c947e5b342d22d644bea9706f"}
Oct 30 13:05:26.092994 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:26.092964    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"
Oct 30 13:05:26.219313 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:26.219219    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b017696f-0424-4f3b-b1bb-20c6d0997076-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b017696f-0424-4f3b-b1bb-20c6d0997076-maas-api-openshift-ca\") pod \"b017696f-0424-4f3b-b1bb-20c6d0997076\" (UID: \"b017696f-0424-4f3b-b1bb-20c6d0997076\") "
Oct 30 13:05:26.219512 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:26.219457    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b017696f-0424-4f3b-b1bb-20c6d0997076-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b017696f-0424-4f3b-b1bb-20c6d0997076-maas-api-xks-ca\") pod \"b017696f-0424-4f3b-b1bb-20c6d0997076\" (UID: \"b017696f-0424-4f3b-b1bb-20c6d0997076\") "
Oct 30 13:05:26.219606 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:26.219588    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b017696f-0424-4f3b-b1bb-20c6d0997076-kube-api-access-v2kfm\" (UniqueName: \"kubernetes.io/projected/b017696f-0424-4f3b-b1bb-20c6d0997076-kube-api-access-v2kfm\") pod \"b017696f-0424-4f3b-b1bb-20c6d0997076\" (UID: \"b017696f-0424-4f3b-b1bb-20c6d0997076\") "
Oct 30 13:05:26.219662 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:26.219621    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b017696f-0424-4f3b-b1bb-20c6d0997076-maas-api-openshift-ca" pod "b017696f-0424-4f3b-b1bb-20c6d0997076" (UID: "b017696f-0424-4f3b-b1bb-20c6d0997076"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:26.220144 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:26.220124    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b017696f-0424-4f3b-b1bb-20c6d0997076-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:26.221681 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:26.221655    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b017696f-0424-4f3b-b1bb-20c6d0997076-maas-api-xks-ca" pod "b017696f-0424-4f3b-b1bb-20c6d0997076" (UID: "b017696f-0424-4f3b-b1bb-20c6d0997076"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:26.221773 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:26.221732    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b017696f-0424-4f3b-b1bb-20c6d0997076-kube-api-access-v2kfm" pod "b017696f-0424-4f3b-b1bb-20c6d0997076" (UID: "b017696f-0424-4f3b-b1bb-20c6d0997076"). InnerVolumeSpecName "kube-api-access-v2kfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:26.321959 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:26.321909    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b017696f-0424-4f3b-b1bb-20c6d0997076-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:26.321959 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:26.321954    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2kfm\" (UniqueName: \"kubernetes.io/projected/b017696f-0424-4f3b-b1bb-20c6d0997076-kube-api-access-v2kfm\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:26.916745 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:26.916714    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2" event={"ID":"b017696f-0424-4f3b-b1bb-20c6d0997076","Type":"ContainerDied","Data":"e99280d4e3904fa4a93270e7ebce253b50401ae05795139d849672dc384d75cf"}
Oct 30 13:05:26.917024 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:26.916766    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e99280d4e3904fa4a93270e7ebce253b50401ae05795139d849672dc384d75cf"
Oct 30 13:05:26.917024 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:26.916726    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"
Oct 30 13:05:27.328952 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.328921    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:05:27.438862 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.438826    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-workload-socket\") pod \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") "
Oct 30 13:05:27.438862 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.438861    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-credential-socket\") pod \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") "
Oct 30 13:05:27.439123 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.438976    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-envoy\") pod \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") "
Oct 30 13:05:27.439123 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.439071    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-kube-api-access-ndpgl\" (UniqueName: \"kubernetes.io/projected/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-kube-api-access-ndpgl\") pod \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") "
Oct 30 13:05:27.439123 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.439108    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-podinfo\") pod \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") "
Oct 30 13:05:27.439294 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.439128    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-workload-socket" pod "4600132d-6b1f-4f1e-bc28-76e86ee9b25f" (UID: "4600132d-6b1f-4f1e-bc28-76e86ee9b25f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:27.439294 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.439165    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-data\") pod \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") "
Oct 30 13:05:27.439294 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.439223    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istiod-ca-cert\") pod \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") "
Oct 30 13:05:27.439294 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.439250    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-credential-socket" pod "4600132d-6b1f-4f1e-bc28-76e86ee9b25f" (UID: "4600132d-6b1f-4f1e-bc28-76e86ee9b25f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:27.439294 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.439274    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-workload-certs\") pod \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") "
Oct 30 13:05:27.439588 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.439377    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-token\" (UniqueName: \"kubernetes.io/projected/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-token\") pod \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\" (UID: \"4600132d-6b1f-4f1e-bc28-76e86ee9b25f\") "
Oct 30 13:05:27.439718 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.439697    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-data" pod "4600132d-6b1f-4f1e-bc28-76e86ee9b25f" (UID: "4600132d-6b1f-4f1e-bc28-76e86ee9b25f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:27.439860 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.439828    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-workload-certs" pod "4600132d-6b1f-4f1e-bc28-76e86ee9b25f" (UID: "4600132d-6b1f-4f1e-bc28-76e86ee9b25f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:27.440084 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.440034    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istiod-ca-cert" pod "4600132d-6b1f-4f1e-bc28-76e86ee9b25f" (UID: "4600132d-6b1f-4f1e-bc28-76e86ee9b25f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:27.440464 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.440443    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:27.440566 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.440468    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:27.440566 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.440485    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:27.440566 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.440498    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:27.440566 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.440513    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:27.441842 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.441814    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-kube-api-access-ndpgl" pod "4600132d-6b1f-4f1e-bc28-76e86ee9b25f" (UID: "4600132d-6b1f-4f1e-bc28-76e86ee9b25f"). InnerVolumeSpecName "kube-api-access-ndpgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:27.442242 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.441958    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-podinfo" pod "4600132d-6b1f-4f1e-bc28-76e86ee9b25f" (UID: "4600132d-6b1f-4f1e-bc28-76e86ee9b25f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:05:27.442242 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.442002    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-envoy" pod "4600132d-6b1f-4f1e-bc28-76e86ee9b25f" (UID: "4600132d-6b1f-4f1e-bc28-76e86ee9b25f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:27.442338 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.442259    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-token" pod "4600132d-6b1f-4f1e-bc28-76e86ee9b25f" (UID: "4600132d-6b1f-4f1e-bc28-76e86ee9b25f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:27.541567 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.541541    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:27.541693 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.541569    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndpgl\" (UniqueName: \"kubernetes.io/projected/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-kube-api-access-ndpgl\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:27.541693 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.541580    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:27.541693 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.541588    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4600132d-6b1f-4f1e-bc28-76e86ee9b25f-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:27.926301 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.926206    5837 generic.go:356] "Generic (PLEG): container finished" podID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerID="1f9605a12a898d1943bd473ac3272ae1ca6ccb723afdacee30e6e0a0b36de7a9" exitCode=0
Oct 30 13:05:27.926301 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.926264    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"
Oct 30 13:05:27.926582 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.926299    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw" event={"ID":"4600132d-6b1f-4f1e-bc28-76e86ee9b25f","Type":"ContainerDied","Data":"1f9605a12a898d1943bd473ac3272ae1ca6ccb723afdacee30e6e0a0b36de7a9"}
Oct 30 13:05:27.926582 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.926351    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw" event={"ID":"4600132d-6b1f-4f1e-bc28-76e86ee9b25f","Type":"ContainerDied","Data":"763b3406cbf78fb6d57bb7dcd5bf668687d35fa6159f8490b5031c8ad11397b6"}
Oct 30 13:05:27.926582 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.926375    5837 scope.go:122] "RemoveContainer" containerID="1f9605a12a898d1943bd473ac3272ae1ca6ccb723afdacee30e6e0a0b36de7a9"
Oct 30 13:05:27.937544 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.937524    5837 scope.go:122] "RemoveContainer" containerID="3f9214571706bed6ecb5f49e021aab15e8cc200c98667c5de21debf8fc6aec6f"
Oct 30 13:05:27.947998 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.947981    5837 scope.go:122] "RemoveContainer" containerID="1f9605a12a898d1943bd473ac3272ae1ca6ccb723afdacee30e6e0a0b36de7a9"
Oct 30 13:05:27.948268 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:05:27.948247    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9605a12a898d1943bd473ac3272ae1ca6ccb723afdacee30e6e0a0b36de7a9\": container with ID starting with 1f9605a12a898d1943bd473ac3272ae1ca6ccb723afdacee30e6e0a0b36de7a9 not found: ID does not exist" containerID="1f9605a12a898d1943bd473ac3272ae1ca6ccb723afdacee30e6e0a0b36de7a9"
Oct 30 13:05:27.948346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.948277    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9605a12a898d1943bd473ac3272ae1ca6ccb723afdacee30e6e0a0b36de7a9"} err="failed to get container status \"1f9605a12a898d1943bd473ac3272ae1ca6ccb723afdacee30e6e0a0b36de7a9\": rpc error: code = NotFound desc = could not find container \"1f9605a12a898d1943bd473ac3272ae1ca6ccb723afdacee30e6e0a0b36de7a9\": container with ID starting with 1f9605a12a898d1943bd473ac3272ae1ca6ccb723afdacee30e6e0a0b36de7a9 not found: ID does not exist"
Oct 30 13:05:27.948346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.948294    5837 scope.go:122] "RemoveContainer" containerID="3f9214571706bed6ecb5f49e021aab15e8cc200c98667c5de21debf8fc6aec6f"
Oct 30 13:05:27.948580 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:05:27.948548    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9214571706bed6ecb5f49e021aab15e8cc200c98667c5de21debf8fc6aec6f\": container with ID starting with 3f9214571706bed6ecb5f49e021aab15e8cc200c98667c5de21debf8fc6aec6f not found: ID does not exist" containerID="3f9214571706bed6ecb5f49e021aab15e8cc200c98667c5de21debf8fc6aec6f"
Oct 30 13:05:27.948682 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.948582    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9214571706bed6ecb5f49e021aab15e8cc200c98667c5de21debf8fc6aec6f"} err="failed to get container status \"3f9214571706bed6ecb5f49e021aab15e8cc200c98667c5de21debf8fc6aec6f\": rpc error: code = NotFound desc = could not find container \"3f9214571706bed6ecb5f49e021aab15e8cc200c98667c5de21debf8fc6aec6f\": container with ID starting with 3f9214571706bed6ecb5f49e021aab15e8cc200c98667c5de21debf8fc6aec6f not found: ID does not exist"
Oct 30 13:05:27.991933 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:27.991900    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"]
Oct 30 13:05:28.003949 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.003917    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ba9a8c-openshift-default-54b94d8bbf-m87tw"]
Oct 30 13:05:28.726712 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.726689    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"
Oct 30 13:05:28.875547 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.875506    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56eb33e2-cc60-4372-8b87-e369b9c12140-kube-api-access-74g64\" (UniqueName: \"kubernetes.io/projected/56eb33e2-cc60-4372-8b87-e369b9c12140-kube-api-access-74g64\") pod \"56eb33e2-cc60-4372-8b87-e369b9c12140\" (UID: \"56eb33e2-cc60-4372-8b87-e369b9c12140\") "
Oct 30 13:05:28.875749 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.875576    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56eb33e2-cc60-4372-8b87-e369b9c12140-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56eb33e2-cc60-4372-8b87-e369b9c12140-plugins-config-volume\") pod \"56eb33e2-cc60-4372-8b87-e369b9c12140\" (UID: \"56eb33e2-cc60-4372-8b87-e369b9c12140\") "
Oct 30 13:05:28.876940 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.876225    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56eb33e2-cc60-4372-8b87-e369b9c12140-plugins-config-volume" pod "56eb33e2-cc60-4372-8b87-e369b9c12140" (UID: "56eb33e2-cc60-4372-8b87-e369b9c12140"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:28.876940 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.876907    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56eb33e2-cc60-4372-8b87-e369b9c12140-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:28.878240 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.878216    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56eb33e2-cc60-4372-8b87-e369b9c12140-kube-api-access-74g64" pod "56eb33e2-cc60-4372-8b87-e369b9c12140" (UID: "56eb33e2-cc60-4372-8b87-e369b9c12140"). InnerVolumeSpecName "kube-api-access-74g64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:28.938657 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.938572    5837 generic.go:356] "Generic (PLEG): container finished" podID="56eb33e2-cc60-4372-8b87-e369b9c12140" containerID="f275cd314954c7ce0d23fe70c38ad6bd00164a00b340349eebd7a3ee40ad2d1a" exitCode=0
Oct 30 13:05:28.938657 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.938630    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"
Oct 30 13:05:28.938860 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.938665    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk" event={"ID":"56eb33e2-cc60-4372-8b87-e369b9c12140","Type":"ContainerDied","Data":"f275cd314954c7ce0d23fe70c38ad6bd00164a00b340349eebd7a3ee40ad2d1a"}
Oct 30 13:05:28.938860 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.938717    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk" event={"ID":"56eb33e2-cc60-4372-8b87-e369b9c12140","Type":"ContainerDied","Data":"95721000284dd9ee3010aa91b54d832e36e1e662c55381d2a7a58f646ce37c81"}
Oct 30 13:05:28.938860 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.938738    5837 scope.go:122] "RemoveContainer" containerID="f275cd314954c7ce0d23fe70c38ad6bd00164a00b340349eebd7a3ee40ad2d1a"
Oct 30 13:05:28.949328 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.949308    5837 scope.go:122] "RemoveContainer" containerID="f275cd314954c7ce0d23fe70c38ad6bd00164a00b340349eebd7a3ee40ad2d1a"
Oct 30 13:05:28.949631 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:05:28.949614    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f275cd314954c7ce0d23fe70c38ad6bd00164a00b340349eebd7a3ee40ad2d1a\": container with ID starting with f275cd314954c7ce0d23fe70c38ad6bd00164a00b340349eebd7a3ee40ad2d1a not found: ID does not exist" containerID="f275cd314954c7ce0d23fe70c38ad6bd00164a00b340349eebd7a3ee40ad2d1a"
Oct 30 13:05:28.949687 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.949638    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f275cd314954c7ce0d23fe70c38ad6bd00164a00b340349eebd7a3ee40ad2d1a"} err="failed to get container status \"f275cd314954c7ce0d23fe70c38ad6bd00164a00b340349eebd7a3ee40ad2d1a\": rpc error: code = NotFound desc = could not find container \"f275cd314954c7ce0d23fe70c38ad6bd00164a00b340349eebd7a3ee40ad2d1a\": container with ID starting with f275cd314954c7ce0d23fe70c38ad6bd00164a00b340349eebd7a3ee40ad2d1a not found: ID does not exist"
Oct 30 13:05:28.987489 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.987459    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74g64\" (UniqueName: \"kubernetes.io/projected/56eb33e2-cc60-4372-8b87-e369b9c12140-kube-api-access-74g64\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:28.998761 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:28.998733    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"]
Oct 30 13:05:29.002445 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:29.002415    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-105228-6f4847cbbc-9zjnk"]
Oct 30 13:05:29.606999 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:29.606966    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" path="/var/lib/kubelet/pods/4600132d-6b1f-4f1e-bc28-76e86ee9b25f/volumes"
Oct 30 13:05:29.607425 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:29.607389    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56eb33e2-cc60-4372-8b87-e369b9c12140" path="/var/lib/kubelet/pods/56eb33e2-cc60-4372-8b87-e369b9c12140/volumes"
Oct 30 13:05:30.028974 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:30.028947    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"]
Oct 30 13:05:30.082803 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:30.082757    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cdd1d1c-6p9zf"]
Oct 30 13:05:31.606791 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:31.606764    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73361c80-e0ae-4807-ae3c-e5a64d2f18fe" path="/var/lib/kubelet/pods/73361c80-e0ae-4807-ae3c-e5a64d2f18fe/volumes"
Oct 30 13:05:32.980369 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:32.980337    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"]
Oct 30 13:05:33.018626 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:33.018596    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"]
Oct 30 13:05:36.708675 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:36.708644    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"]
Oct 30 13:05:36.720448 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:36.720413    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-105228-w7xhh"]
Oct 30 13:05:37.607641 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:37.607612    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2728869d-2a7c-4530-8f53-93309ebf4230" path="/var/lib/kubelet/pods/2728869d-2a7c-4530-8f53-93309ebf4230/volumes"
Oct 30 13:05:37.981622 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:37.981561    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4" podUID="2a03f1bb-dfc9-4e35-9723-8ec01432a544" containerName="payload-processing" containerID="cri-o://e8cf75a19d830090f3ceecafe2c2eae809fda74d95e52e998b427ea739c2f404" gracePeriod=25
Oct 30 13:05:38.019329 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:38.019298    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp" podUID="188cba42-8f91-4d73-a8be-858791be060b" containerName="payload-pre-processing" containerID="cri-o://c6a0aa91b41059ea8c80ff1efba24d94659cb828d37b1ceb1633734ed1f5d5e3" gracePeriod=25
Oct 30 13:05:38.345307 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:38.345284    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"
Oct 30 13:05:38.348740 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:38.348720    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"
Oct 30 13:05:38.443495 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:38.443461    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188cba42-8f91-4d73-a8be-858791be060b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188cba42-8f91-4d73-a8be-858791be060b-plugins-config-volume\") pod \"188cba42-8f91-4d73-a8be-858791be060b\" (UID: \"188cba42-8f91-4d73-a8be-858791be060b\") "
Oct 30 13:05:38.443683 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:38.443590    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a03f1bb-dfc9-4e35-9723-8ec01432a544-kube-api-access-wd578\" (UniqueName: \"kubernetes.io/projected/2a03f1bb-dfc9-4e35-9723-8ec01432a544-kube-api-access-wd578\") pod \"2a03f1bb-dfc9-4e35-9723-8ec01432a544\" (UID: \"2a03f1bb-dfc9-4e35-9723-8ec01432a544\") "
Oct 30 13:05:38.443753 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:38.443684    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188cba42-8f91-4d73-a8be-858791be060b-kube-api-access-bbntz\" (UniqueName: \"kubernetes.io/projected/188cba42-8f91-4d73-a8be-858791be060b-kube-api-access-bbntz\") pod \"188cba42-8f91-4d73-a8be-858791be060b\" (UID: \"188cba42-8f91-4d73-a8be-858791be060b\") "
Oct 30 13:05:38.443813 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:38.443797    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a03f1bb-dfc9-4e35-9723-8ec01432a544-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a03f1bb-dfc9-4e35-9723-8ec01432a544-plugins-config-volume\") pod \"2a03f1bb-dfc9-4e35-9723-8ec01432a544\" (UID: \"2a03f1bb-dfc9-4e35-9723-8ec01432a544\") "
Oct 30 13:05:38.443944 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:38.443924    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188cba42-8f91-4d73-a8be-858791be060b-plugins-config-volume" pod "188cba42-8f91-4d73-a8be-858791be060b" (UID: "188cba42-8f91-4d73-a8be-858791be060b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:38.444622 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:38.444598    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188cba42-8f91-4d73-a8be-858791be060b-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:38.444814 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:38.444627    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a03f1bb-dfc9-4e35-9723-8ec01432a544-plugins-config-volume" pod "2a03f1bb-dfc9-4e35-9723-8ec01432a544" (UID: "2a03f1bb-dfc9-4e35-9723-8ec01432a544"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:38.446316 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:38.446287    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a03f1bb-dfc9-4e35-9723-8ec01432a544-kube-api-access-wd578" pod "2a03f1bb-dfc9-4e35-9723-8ec01432a544" (UID: "2a03f1bb-dfc9-4e35-9723-8ec01432a544"). InnerVolumeSpecName "kube-api-access-wd578". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:38.446665 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:38.446644    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188cba42-8f91-4d73-a8be-858791be060b-kube-api-access-bbntz" pod "188cba42-8f91-4d73-a8be-858791be060b" (UID: "188cba42-8f91-4d73-a8be-858791be060b"). InnerVolumeSpecName "kube-api-access-bbntz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:38.545836 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:38.545760    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbntz\" (UniqueName: \"kubernetes.io/projected/188cba42-8f91-4d73-a8be-858791be060b-kube-api-access-bbntz\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:38.545836 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:38.545788    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a03f1bb-dfc9-4e35-9723-8ec01432a544-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:38.545836 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:38.545798    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd578\" (UniqueName: \"kubernetes.io/projected/2a03f1bb-dfc9-4e35-9723-8ec01432a544-kube-api-access-wd578\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:39.029219 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.029190    5837 generic.go:356] "Generic (PLEG): container finished" podID="188cba42-8f91-4d73-a8be-858791be060b" containerID="c6a0aa91b41059ea8c80ff1efba24d94659cb828d37b1ceb1633734ed1f5d5e3" exitCode=0
Oct 30 13:05:39.029644 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.029251    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"
Oct 30 13:05:39.029644 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.029274    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp" event={"ID":"188cba42-8f91-4d73-a8be-858791be060b","Type":"ContainerDied","Data":"c6a0aa91b41059ea8c80ff1efba24d94659cb828d37b1ceb1633734ed1f5d5e3"}
Oct 30 13:05:39.029644 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.029337    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp" event={"ID":"188cba42-8f91-4d73-a8be-858791be060b","Type":"ContainerDied","Data":"b042e0e1bbcd5f37bf15fefb9149104bbeadc33d31cbf3ffc838a5edb2696c38"}
Oct 30 13:05:39.029644 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.029368    5837 scope.go:122] "RemoveContainer" containerID="c6a0aa91b41059ea8c80ff1efba24d94659cb828d37b1ceb1633734ed1f5d5e3"
Oct 30 13:05:39.031229 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.031204    5837 generic.go:356] "Generic (PLEG): container finished" podID="2a03f1bb-dfc9-4e35-9723-8ec01432a544" containerID="e8cf75a19d830090f3ceecafe2c2eae809fda74d95e52e998b427ea739c2f404" exitCode=0
Oct 30 13:05:39.031327 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.031259    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4" event={"ID":"2a03f1bb-dfc9-4e35-9723-8ec01432a544","Type":"ContainerDied","Data":"e8cf75a19d830090f3ceecafe2c2eae809fda74d95e52e998b427ea739c2f404"}
Oct 30 13:05:39.031327 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.031302    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4" event={"ID":"2a03f1bb-dfc9-4e35-9723-8ec01432a544","Type":"ContainerDied","Data":"dda61d42b5129e5749b7fee5c2fd235d3e089e6c6cf5850a2acfb1d611e8caa2"}
Oct 30 13:05:39.031440 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.031263    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"
Oct 30 13:05:39.041817 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.041801    5837 scope.go:122] "RemoveContainer" containerID="c6a0aa91b41059ea8c80ff1efba24d94659cb828d37b1ceb1633734ed1f5d5e3"
Oct 30 13:05:39.042036 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:05:39.042020    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a0aa91b41059ea8c80ff1efba24d94659cb828d37b1ceb1633734ed1f5d5e3\": container with ID starting with c6a0aa91b41059ea8c80ff1efba24d94659cb828d37b1ceb1633734ed1f5d5e3 not found: ID does not exist" containerID="c6a0aa91b41059ea8c80ff1efba24d94659cb828d37b1ceb1633734ed1f5d5e3"
Oct 30 13:05:39.042085 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.042043    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a0aa91b41059ea8c80ff1efba24d94659cb828d37b1ceb1633734ed1f5d5e3"} err="failed to get container status \"c6a0aa91b41059ea8c80ff1efba24d94659cb828d37b1ceb1633734ed1f5d5e3\": rpc error: code = NotFound desc = could not find container \"c6a0aa91b41059ea8c80ff1efba24d94659cb828d37b1ceb1633734ed1f5d5e3\": container with ID starting with c6a0aa91b41059ea8c80ff1efba24d94659cb828d37b1ceb1633734ed1f5d5e3 not found: ID does not exist"
Oct 30 13:05:39.042085 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.042059    5837 scope.go:122] "RemoveContainer" containerID="e8cf75a19d830090f3ceecafe2c2eae809fda74d95e52e998b427ea739c2f404"
Oct 30 13:05:39.054318 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.054298    5837 scope.go:122] "RemoveContainer" containerID="e8cf75a19d830090f3ceecafe2c2eae809fda74d95e52e998b427ea739c2f404"
Oct 30 13:05:39.054654 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:05:39.054634    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cf75a19d830090f3ceecafe2c2eae809fda74d95e52e998b427ea739c2f404\": container with ID starting with e8cf75a19d830090f3ceecafe2c2eae809fda74d95e52e998b427ea739c2f404 not found: ID does not exist" containerID="e8cf75a19d830090f3ceecafe2c2eae809fda74d95e52e998b427ea739c2f404"
Oct 30 13:05:39.054715 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.054666    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cf75a19d830090f3ceecafe2c2eae809fda74d95e52e998b427ea739c2f404"} err="failed to get container status \"e8cf75a19d830090f3ceecafe2c2eae809fda74d95e52e998b427ea739c2f404\": rpc error: code = NotFound desc = could not find container \"e8cf75a19d830090f3ceecafe2c2eae809fda74d95e52e998b427ea739c2f404\": container with ID starting with e8cf75a19d830090f3ceecafe2c2eae809fda74d95e52e998b427ea739c2f404 not found: ID does not exist"
Oct 30 13:05:39.091321 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.091292    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"]
Oct 30 13:05:39.103152 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.103097    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1f91c1-6bdd6f76c9-sbvgp"]
Oct 30 13:05:39.129506 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.129463    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"]
Oct 30 13:05:39.133335 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.133310    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1f91c1-6c56bffc56-b9lw4"]
Oct 30 13:05:39.606627 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.606598    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188cba42-8f91-4d73-a8be-858791be060b" path="/var/lib/kubelet/pods/188cba42-8f91-4d73-a8be-858791be060b/volumes"
Oct 30 13:05:39.607119 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:39.607105    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a03f1bb-dfc9-4e35-9723-8ec01432a544" path="/var/lib/kubelet/pods/2a03f1bb-dfc9-4e35-9723-8ec01432a544/volumes"
Oct 30 13:05:43.877382 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:43.877349    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"]
Oct 30 13:05:43.878157 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:43.878123    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy" containerID="cri-o://0cbe8dddddf852091525970918b0aa97613358c180e32f3ac228a7c3bc0e3428" gracePeriod=30
Oct 30 13:05:43.889114 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:43.889088    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"]
Oct 30 13:05:43.889325 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:43.889301    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" podUID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" containerName="istio-proxy" containerID="cri-o://cf30093d1663265d14fa7211b09ca5855854e6943f6bd85a78ac05905f891125" gracePeriod=30
Oct 30 13:05:43.896672 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:43.896645    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"]
Oct 30 13:05:43.896960 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:43.896927    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy" containerID="cri-o://b228ce175ea14df44608df096c649b6efd65de9c4f3e0b595eac7d14405882fa" gracePeriod=30
Oct 30 13:05:45.097224 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:05:45.097191    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:05:47.345598 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:47.345561    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"]
Oct 30 13:05:47.359145 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:47.359109    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f91c1-6r5b2"]
Oct 30 13:05:47.609106 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:47.609028    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b017696f-0424-4f3b-b1bb-20c6d0997076" path="/var/lib/kubelet/pods/b017696f-0424-4f3b-b1bb-20c6d0997076/volumes"
Oct 30 13:05:48.781418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:48.779414    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:05:48.781418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:48.779485    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" podUID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:05:49.340245 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:49.339993    5837 generic.go:356] "Generic (PLEG): container finished" podID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" containerID="cf30093d1663265d14fa7211b09ca5855854e6943f6bd85a78ac05905f891125" exitCode=0
Oct 30 13:05:49.340245 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:49.340115    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" event={"ID":"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd","Type":"ContainerDied","Data":"cf30093d1663265d14fa7211b09ca5855854e6943f6bd85a78ac05905f891125"}
Oct 30 13:05:49.395307 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:49.395079    5837 generic.go:356] "Generic (PLEG): container finished" podID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerID="0cbe8dddddf852091525970918b0aa97613358c180e32f3ac228a7c3bc0e3428" exitCode=0
Oct 30 13:05:49.395307 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:49.395227    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" event={"ID":"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9","Type":"ContainerDied","Data":"0cbe8dddddf852091525970918b0aa97613358c180e32f3ac228a7c3bc0e3428"}
Oct 30 13:05:49.398464 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:49.398236    5837 generic.go:356] "Generic (PLEG): container finished" podID="f8284bf7-b488-4b53-8804-4636dc743903" containerID="b228ce175ea14df44608df096c649b6efd65de9c4f3e0b595eac7d14405882fa" exitCode=0
Oct 30 13:05:49.398464 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:49.398404    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" event={"ID":"f8284bf7-b488-4b53-8804-4636dc743903","Type":"ContainerDied","Data":"b228ce175ea14df44608df096c649b6efd65de9c4f3e0b595eac7d14405882fa"}
Oct 30 13:05:50.260151 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:50.259489    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:50.397925 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:50.388292    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:05:50.397925 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:50.388369    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 13:05:50.454430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:50.449824    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 13:05:50.576602 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:50.561166    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded"
Oct 30 13:05:50.576602 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:50.561544    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 13:05:50.735430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:50.638769    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:50.735430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:50.726882    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.064207    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.071531    5837 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.071842    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.110338    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-workload-socket\") pod \"f8284bf7-b488-4b53-8804-4636dc743903\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") "
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.110530    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-istio-data\") pod \"f8284bf7-b488-4b53-8804-4636dc743903\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") "
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.110627    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8284bf7-b488-4b53-8804-4636dc743903-kube-api-access-pxlj9\" (UniqueName: \"kubernetes.io/projected/f8284bf7-b488-4b53-8804-4636dc743903-kube-api-access-pxlj9\") pod \"f8284bf7-b488-4b53-8804-4636dc743903\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") "
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.110704    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8284bf7-b488-4b53-8804-4636dc743903-istio-token\" (UniqueName: \"kubernetes.io/projected/f8284bf7-b488-4b53-8804-4636dc743903-istio-token\") pod \"f8284bf7-b488-4b53-8804-4636dc743903\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") "
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.110968    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8284bf7-b488-4b53-8804-4636dc743903-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8284bf7-b488-4b53-8804-4636dc743903-istiod-ca-cert\") pod \"f8284bf7-b488-4b53-8804-4636dc743903\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") "
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.111010    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-credential-socket\") pod \"f8284bf7-b488-4b53-8804-4636dc743903\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") "
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.111065    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-istio-envoy\") pod \"f8284bf7-b488-4b53-8804-4636dc743903\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") "
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.111124    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8284bf7-b488-4b53-8804-4636dc743903-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8284bf7-b488-4b53-8804-4636dc743903-istio-podinfo\") pod \"f8284bf7-b488-4b53-8804-4636dc743903\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") "
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.111319    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-workload-certs\") pod \"f8284bf7-b488-4b53-8804-4636dc743903\" (UID: \"f8284bf7-b488-4b53-8804-4636dc743903\") "
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.128814    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-workload-certs" pod "f8284bf7-b488-4b53-8804-4636dc743903" (UID: "f8284bf7-b488-4b53-8804-4636dc743903"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.130206    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-istio-data" pod "f8284bf7-b488-4b53-8804-4636dc743903" (UID: "f8284bf7-b488-4b53-8804-4636dc743903"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.134619    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-workload-socket" pod "f8284bf7-b488-4b53-8804-4636dc743903" (UID: "f8284bf7-b488-4b53-8804-4636dc743903"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:51.228561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.225527    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-credential-socket" pod "f8284bf7-b488-4b53-8804-4636dc743903" (UID: "f8284bf7-b488-4b53-8804-4636dc743903"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:51.415467 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.365192    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:51.415467 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.365231    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:51.415467 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.365302    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:51.415467 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.365327    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:51.415467 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.392730    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8284bf7-b488-4b53-8804-4636dc743903-istiod-ca-cert" pod "f8284bf7-b488-4b53-8804-4636dc743903" (UID: "f8284bf7-b488-4b53-8804-4636dc743903"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:51.415467 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.406539    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:05:51.415467 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.407337    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 13:05:51.631420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.603608    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8284bf7-b488-4b53-8804-4636dc743903-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:51.693358 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.622661    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 13:05:51.694003 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.654105    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:51.824423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.801965    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:51.849509 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.849470    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-vhr7q" podUID="c06bb10a-6170-4c87-82e8-4695cdddc789" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:51.989812 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:51.989773    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-istio-envoy" pod "f8284bf7-b488-4b53-8804-4636dc743903" (UID: "f8284bf7-b488-4b53-8804-4636dc743903"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:52.042486 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.027576    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8284bf7-b488-4b53-8804-4636dc743903-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:52.042486 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.027845    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 13:05:52.068161 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.068123    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:05:52.109448 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.070524    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:52.109448 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.089501    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:52.109448 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.092699    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8284bf7-b488-4b53-8804-4636dc743903-kube-api-access-pxlj9" pod "f8284bf7-b488-4b53-8804-4636dc743903" (UID: "f8284bf7-b488-4b53-8804-4636dc743903"). InnerVolumeSpecName "kube-api-access-pxlj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:52.111714 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.111668    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8284bf7-b488-4b53-8804-4636dc743903-istio-podinfo" pod "f8284bf7-b488-4b53-8804-4636dc743903" (UID: "f8284bf7-b488-4b53-8804-4636dc743903"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:05:52.236489 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.217889    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxlj9\" (UniqueName: \"kubernetes.io/projected/f8284bf7-b488-4b53-8804-4636dc743903-kube-api-access-pxlj9\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:52.236489 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.217920    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8284bf7-b488-4b53-8804-4636dc743903-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:52.278500 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.269425    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"
Oct 30 13:05:52.492476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.491487    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:52.630738 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.630702    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/health\": context deadline exceeded"
Oct 30 13:05:52.669839 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.669806    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:52.670177 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.670150    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26" event={"ID":"f8284bf7-b488-4b53-8804-4636dc743903","Type":"ContainerDied","Data":"eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104"}
Oct 30 13:05:52.670341 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.670324    5837 scope.go:122] "RemoveContainer" containerID="b228ce175ea14df44608df096c649b6efd65de9c4f3e0b595eac7d14405882fa"
Oct 30 13:05:52.709710 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.706606    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:52.746521 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.710494    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 13:05:52.748840 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.748793    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8284bf7-b488-4b53-8804-4636dc743903-istio-token" pod "f8284bf7-b488-4b53-8804-4636dc743903" (UID: "f8284bf7-b488-4b53-8804-4636dc743903"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:52.838845 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.838812    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8284bf7-b488-4b53-8804-4636dc743903-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:52.903791 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.903750    5837 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:52.904059 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:52.904029    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:53.070839 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.033652    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/health\": context deadline exceeded"
Oct 30 13:05:53.070839 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.052111    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 13:05:53.071670 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.071628    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:53.100746 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.098666    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:53.131421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.122178    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 13:05:53.220542 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.220510    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-workload-certs\") pod \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") "
Oct 30 13:05:53.220817 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.220799    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istiod-ca-cert\") pod \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") "
Oct 30 13:05:53.220959 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.220944    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-workload-socket\") pod \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") "
Oct 30 13:05:53.221088 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.221075    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-workload-certs\") pod \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") "
Oct 30 13:05:53.221233 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.221217    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-data\") pod \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") "
Oct 30 13:05:53.221382 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.221368    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-podinfo\") pod \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") "
Oct 30 13:05:53.221557 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.221540    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-kube-api-access-l84kg\" (UniqueName: \"kubernetes.io/projected/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-kube-api-access-l84kg\") pod \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") "
Oct 30 13:05:53.221677 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.221663    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-token\" (UniqueName: \"kubernetes.io/projected/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-token\") pod \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") "
Oct 30 13:05:53.221778 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.221764    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-envoy\") pod \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") "
Oct 30 13:05:53.227426 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.221923    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-workload-socket\") pod \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") "
Oct 30 13:05:53.227770 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.227746    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-credential-socket\") pod \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") "
Oct 30 13:05:53.227939 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.227923    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-envoy\") pod \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") "
Oct 30 13:05:53.228079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.228063    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-podinfo\") pod \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") "
Oct 30 13:05:53.228205 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.228188    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-credential-socket\") pod \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") "
Oct 30 13:05:53.228381 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.228358    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-kube-api-access-bpmlk\" (UniqueName: \"kubernetes.io/projected/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-kube-api-access-bpmlk\") pod \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") "
Oct 30 13:05:53.228559 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.228540    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istiod-ca-cert\") pod \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") "
Oct 30 13:05:53.228705 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.228690    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-data\") pod \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\" (UID: \"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9\") "
Oct 30 13:05:53.228906 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.228888    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-token\" (UniqueName: \"kubernetes.io/projected/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-token\") pod \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\" (UID: \"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd\") "
Oct 30 13:05:53.267856 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.267815    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-workload-socket" pod "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" (UID: "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:53.268378 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.268350    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-workload-certs" pod "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" (UID: "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:53.272608 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.272576    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istiod-ca-cert" pod "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" (UID: "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:53.277813 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.277785    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-workload-socket" pod "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" (UID: "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:53.307506 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.277996    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-credential-socket" pod "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" (UID: "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:53.345424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.333560    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-envoy" pod "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" (UID: "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:53.349485 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.341936    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.374543 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.374509    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.378424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.374759    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.403496 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.403458    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.403759 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.403735    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.480421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.477880    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istiod-ca-cert" pod "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" (UID: "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:53.481183 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.480958    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-workload-certs" pod "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" (UID: "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:53.516479 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.503491    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-credential-socket" pod "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" (UID: "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:53.528996 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.517513    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-data" pod "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" (UID: "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:53.569428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.569099    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-data" pod "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" (UID: "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:53.575209 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.575173    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.645948 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.641524    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.645948 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.641576    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.645948 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.641592    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.645948 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.641606    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.645948 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.641622    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.731489 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.675568    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:53.781435 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.692316    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-token" pod "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" (UID: "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:53.781435 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.702991    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-envoy" pod "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" (UID: "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:53.781435 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.747866    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-podinfo" pod "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" (UID: "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:05:53.781435 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.726973    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:53.791449 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.706360    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"
Oct 30 13:05:53.791449 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.704025    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"
Oct 30 13:05:53.992291 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.990809    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:54.027824 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.992933    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.027824 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.992960    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.027824 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:53.992977    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.035605 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.029778    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-kube-api-access-bpmlk" pod "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" (UID: "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd"). InnerVolumeSpecName "kube-api-access-bpmlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:54.059551 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.053830    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-kube-api-access-l84kg" pod "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" (UID: "b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9"). InnerVolumeSpecName "kube-api-access-l84kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:54.184445 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.094360    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l84kg\" (UniqueName: \"kubernetes.io/projected/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9-kube-api-access-l84kg\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.184445 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.094414    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpmlk\" (UniqueName: \"kubernetes.io/projected/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-kube-api-access-bpmlk\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.184445 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.122895    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": context deadline exceeded"
Oct 30 13:05:54.184445 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.126472    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/ready\": context deadline exceeded"
Oct 30 13:05:54.184445 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.181602    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-token" pod "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" (UID: "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:54.243616 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.229344    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.361854 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.361689    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-podinfo" pod "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" (UID: "d2c2ed7b-35e0-4696-ad52-72cc6c2472cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:05:54.418103 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.418060    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg" event={"ID":"d2c2ed7b-35e0-4696-ad52-72cc6c2472cd","Type":"ContainerDied","Data":"03a51cc1a46cf41d3769dfa07c3fd9928d533b29bd7dbff9db840d48f1894dcd"}
Oct 30 13:05:54.446480 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.446454    5837 scope.go:122] "RemoveContainer" containerID="cf30093d1663265d14fa7211b09ca5855854e6943f6bd85a78ac05905f891125"
Oct 30 13:05:54.448011 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.446385    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr" event={"ID":"b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9","Type":"ContainerDied","Data":"a1bfa8cb0027e0d73f093d4b3228939429cef23c01dd3af949ad2f3774c58e4e"}
Oct 30 13:05:54.460018 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.459901    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"]
Oct 30 13:05:54.466616 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.466495    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rqh26"]
Oct 30 13:05:54.571790 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.571454    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"]
Oct 30 13:05:54.712432 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.703164    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.731423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.715606    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": context deadline exceeded"
Oct 30 13:05:54.807571 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.807530    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 13:05:54.808919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.808862    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:05:54.843768 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:54.843679    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-hl4xr"]
Oct 30 13:05:55.021486 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.021450    5837 scope.go:122] "RemoveContainer" containerID="0cbe8dddddf852091525970918b0aa97613358c180e32f3ac228a7c3bc0e3428"
Oct 30 13:05:55.063444 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.051207    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:05:55.279420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.278323    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:55.279420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.278569    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:55.353726 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.353697    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:05:55.465482 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.464457    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"]
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466300    5837 cpu_manager.go:414] "containerMap: removing container" podUID="188cba42-8f91-4d73-a8be-858791be060b" containerName="payload-pre-processing"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466329    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188cba42-8f91-4d73-a8be-858791be060b" containerName="payload-pre-processing"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466372    5837 cpu_manager.go:414] "containerMap: removing container" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466381    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466424    5837 cpu_manager.go:414] "containerMap: removing container" podUID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" containerName="istio-proxy"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466433    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" containerName="istio-proxy"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466481    5837 cpu_manager.go:414] "containerMap: removing container" podUID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerName="istio-proxy"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466489    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerName="istio-proxy"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466512    5837 cpu_manager.go:414] "containerMap: removing container" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466520    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466558    5837 cpu_manager.go:414] "containerMap: removing container" podUID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerName="istio-proxy"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466566    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerName="istio-proxy"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466644    5837 cpu_manager.go:414] "containerMap: removing container" podUID="56eb33e2-cc60-4372-8b87-e369b9c12140" containerName="payload-pre-processing"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466653    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56eb33e2-cc60-4372-8b87-e369b9c12140" containerName="payload-pre-processing"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466679    5837 cpu_manager.go:414] "containerMap: removing container" podUID="b017696f-0424-4f3b-b1bb-20c6d0997076" containerName="revoke-keys"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466688    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b017696f-0424-4f3b-b1bb-20c6d0997076" containerName="revoke-keys"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466708    5837 cpu_manager.go:414] "containerMap: removing container" podUID="2a03f1bb-dfc9-4e35-9723-8ec01432a544" containerName="payload-processing"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.466716    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a03f1bb-dfc9-4e35-9723-8ec01432a544" containerName="payload-processing"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.467091    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="56eb33e2-cc60-4372-8b87-e369b9c12140" containerName="payload-pre-processing"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.467120    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b017696f-0424-4f3b-b1bb-20c6d0997076" containerName="revoke-keys"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.467132    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a03f1bb-dfc9-4e35-9723-8ec01432a544" containerName="payload-processing"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.467152    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="188cba42-8f91-4d73-a8be-858791be060b" containerName="payload-pre-processing"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.467164    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" containerName="istio-proxy"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.467173    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" containerName="istio-proxy"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.467201    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerName="istio-proxy"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.467219    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8284bf7-b488-4b53-8804-4636dc743903" containerName="istio-proxy"
Oct 30 13:05:55.473422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.467239    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4600132d-6b1f-4f1e-bc28-76e86ee9b25f" containerName="istio-proxy"
Oct 30 13:05:55.694429 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.650668    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"]
Oct 30 13:05:55.694429 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.671447    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"
Oct 30 13:05:55.750424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.746049    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:05:55.750424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.747018    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:05:55.750424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.747148    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:05:55.822357 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.822312    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:55.833638 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.833586    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 13:05:55.871572 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.863598    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh9d4\" (UniqueName: \"kubernetes.io/projected/797bf104-289c-4c41-97b8-9413b316dba2-kube-api-access-dh9d4\") pod \"maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729\" (UID: \"797bf104-289c-4c41-97b8-9413b316dba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"
Oct 30 13:05:55.871572 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.864132    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/797bf104-289c-4c41-97b8-9413b316dba2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729\" (UID: \"797bf104-289c-4c41-97b8-9413b316dba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"
Oct 30 13:05:55.871572 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.864174    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/797bf104-289c-4c41-97b8-9413b316dba2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729\" (UID: \"797bf104-289c-4c41-97b8-9413b316dba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"
Oct 30 13:05:55.899439 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:55.891789    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:05:56.069476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.045954    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/797bf104-289c-4c41-97b8-9413b316dba2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729\" (UID: \"797bf104-289c-4c41-97b8-9413b316dba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"
Oct 30 13:05:56.101739 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.101697    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/797bf104-289c-4c41-97b8-9413b316dba2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729\" (UID: \"797bf104-289c-4c41-97b8-9413b316dba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"
Oct 30 13:05:56.102149 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.102120    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9d4\" (UniqueName: \"kubernetes.io/projected/797bf104-289c-4c41-97b8-9413b316dba2-kube-api-access-dh9d4\") pod \"maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729\" (UID: \"797bf104-289c-4c41-97b8-9413b316dba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"
Oct 30 13:05:56.102966 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.091661    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9" path="/var/lib/kubelet/pods/b107ba3d-a470-4c0b-a3bf-5b57ecafe0d9/volumes"
Oct 30 13:05:56.124879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.124850    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8284bf7-b488-4b53-8804-4636dc743903" path="/var/lib/kubelet/pods/f8284bf7-b488-4b53-8804-4636dc743903/volumes"
Oct 30 13:05:56.125766 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.125736    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"]
Oct 30 13:05:56.126323 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.126293    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerName="istio-proxy" containerID="cri-o://06a664ce2615e78d052b6912976f70f30db9f34c5b2cbf3c006260482514bec4" gracePeriod=30
Oct 30 13:05:56.127347 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.127302    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:56.130774 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.073364    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/797bf104-289c-4c41-97b8-9413b316dba2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729\" (UID: \"797bf104-289c-4c41-97b8-9413b316dba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"
Oct 30 13:05:56.241793 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.241762    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"]
Oct 30 13:05:56.245256 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.245227    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh9d4\" (UniqueName: \"kubernetes.io/projected/797bf104-289c-4c41-97b8-9413b316dba2-kube-api-access-dh9d4\") pod \"maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729\" (UID: \"797bf104-289c-4c41-97b8-9413b316dba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"
Oct 30 13:05:56.369987 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.369889    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:56.466133 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.466051    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/797bf104-289c-4c41-97b8-9413b316dba2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729\" (UID: \"797bf104-289c-4c41-97b8-9413b316dba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"
Oct 30 13:05:56.472472 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.470454    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-b7spg"]
Oct 30 13:05:56.692482 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.688249    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"
Oct 30 13:05:56.795617 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.782253    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:05:56.841699 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.831763    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:56.841699 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:56.840875    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:57.036318 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:57.036053    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 13:05:57.038092 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:57.038069    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 13:05:57.052864 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:57.052840    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:05:57.125026 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:05:57.118907    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice/crio-eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c2ed7b_35e0_4696_ad52_72cc6c2472cd.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:05:57.418825 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:57.418231    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"]
Oct 30 13:05:57.582317 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:57.582282    5837 scope.go:122] "RemoveContainer" containerID="8548f354fb10acbfb237eb405bce8657a71c797d9ea31aa6e72e368dac6ed14f"
Oct 30 13:05:57.638551 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:57.635122    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2c2ed7b-35e0-4696-ad52-72cc6c2472cd" path="/var/lib/kubelet/pods/d2c2ed7b-35e0-4696-ad52-72cc6c2472cd/volumes"
Oct 30 13:05:58.002426 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:57.992903    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729" event={"ID":"797bf104-289c-4c41-97b8-9413b316dba2","Type":"ContainerStarted","Data":"0cbd0d7ab8822892df85fb272cc53638b68a8053628ff6d324e45ab476553bc1"}
Oct 30 13:05:58.002426 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:57.992974    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729" event={"ID":"797bf104-289c-4c41-97b8-9413b316dba2","Type":"ContainerStarted","Data":"8ef23a82fabfaf10a7a01ab69859fd6e5bd0464e5e6f51d294b75b988eabc901"}
Oct 30 13:05:58.045661 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:58.045342    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729" podStartSLOduration=4.045321423 podStartE2EDuration="4.045321423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:58.044050994 +0000 UTC m=+1746.568335239" watchObservedRunningTime="2026-10-30 13:05:58.045321423 +0000 UTC m=+1746.569605643"
Oct 30 13:05:59.015804 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:59.015769    5837 generic.go:356] "Generic (PLEG): container finished" podID="797bf104-289c-4c41-97b8-9413b316dba2" containerID="0cbd0d7ab8822892df85fb272cc53638b68a8053628ff6d324e45ab476553bc1" exitCode=0
Oct 30 13:05:59.016216 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:05:59.015848    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729" event={"ID":"797bf104-289c-4c41-97b8-9413b316dba2","Type":"ContainerDied","Data":"0cbd0d7ab8822892df85fb272cc53638b68a8053628ff6d324e45ab476553bc1"}
Oct 30 13:06:00.253486 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:00.253461    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"
Oct 30 13:06:00.377185 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:00.377097    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/797bf104-289c-4c41-97b8-9413b316dba2-kube-api-access-dh9d4\" (UniqueName: \"kubernetes.io/projected/797bf104-289c-4c41-97b8-9413b316dba2-kube-api-access-dh9d4\") pod \"797bf104-289c-4c41-97b8-9413b316dba2\" (UID: \"797bf104-289c-4c41-97b8-9413b316dba2\") "
Oct 30 13:06:00.377185 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:00.377170    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/797bf104-289c-4c41-97b8-9413b316dba2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/797bf104-289c-4c41-97b8-9413b316dba2-maas-api-openshift-ca\") pod \"797bf104-289c-4c41-97b8-9413b316dba2\" (UID: \"797bf104-289c-4c41-97b8-9413b316dba2\") "
Oct 30 13:06:00.377464 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:00.377288    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/797bf104-289c-4c41-97b8-9413b316dba2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/797bf104-289c-4c41-97b8-9413b316dba2-maas-api-xks-ca\") pod \"797bf104-289c-4c41-97b8-9413b316dba2\" (UID: \"797bf104-289c-4c41-97b8-9413b316dba2\") "
Oct 30 13:06:00.377612 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:00.377587    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797bf104-289c-4c41-97b8-9413b316dba2-maas-api-openshift-ca" pod "797bf104-289c-4c41-97b8-9413b316dba2" (UID: "797bf104-289c-4c41-97b8-9413b316dba2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:00.378038 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:00.378018    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/797bf104-289c-4c41-97b8-9413b316dba2-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:00.431417 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:00.430147    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/797bf104-289c-4c41-97b8-9413b316dba2-maas-api-xks-ca" pod "797bf104-289c-4c41-97b8-9413b316dba2" (UID: "797bf104-289c-4c41-97b8-9413b316dba2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:00.431417 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:00.430326    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797bf104-289c-4c41-97b8-9413b316dba2-kube-api-access-dh9d4" pod "797bf104-289c-4c41-97b8-9413b316dba2" (UID: "797bf104-289c-4c41-97b8-9413b316dba2"). InnerVolumeSpecName "kube-api-access-dh9d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:00.479777 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:00.479743    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh9d4\" (UniqueName: \"kubernetes.io/projected/797bf104-289c-4c41-97b8-9413b316dba2-kube-api-access-dh9d4\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:00.479777 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:00.479776    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/797bf104-289c-4c41-97b8-9413b316dba2-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:00.969621 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:00.969544    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice/crio-eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104\": RecentStats: unable to find data in memory cache]"
Oct 30 13:06:01.032236 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.032202    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"
Oct 30 13:06:01.032437 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.032234    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729" event={"ID":"797bf104-289c-4c41-97b8-9413b316dba2","Type":"ContainerDied","Data":"8ef23a82fabfaf10a7a01ab69859fd6e5bd0464e5e6f51d294b75b988eabc901"}
Oct 30 13:06:01.032437 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.032296    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ef23a82fabfaf10a7a01ab69859fd6e5bd0464e5e6f51d294b75b988eabc901"
Oct 30 13:06:01.486992 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.486927    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:06:01.606480 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.606443    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f163e2e9-ace0-44b6-962c-19b61fe31c95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f163e2e9-ace0-44b6-962c-19b61fe31c95-istiod-ca-cert\") pod \"f163e2e9-ace0-44b6-962c-19b61fe31c95\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") "
Oct 30 13:06:01.606659 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.606509    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-podinfo\") pod \"f163e2e9-ace0-44b6-962c-19b61fe31c95\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") "
Oct 30 13:06:01.606809 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.606674    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-data\") pod \"f163e2e9-ace0-44b6-962c-19b61fe31c95\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") "
Oct 30 13:06:01.606809 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.606737    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-token\" (UniqueName: \"kubernetes.io/projected/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-token\") pod \"f163e2e9-ace0-44b6-962c-19b61fe31c95\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") "
Oct 30 13:06:01.606809 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.606773    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-workload-certs\") pod \"f163e2e9-ace0-44b6-962c-19b61fe31c95\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") "
Oct 30 13:06:01.606998 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.606889    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-workload-socket\") pod \"f163e2e9-ace0-44b6-962c-19b61fe31c95\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") "
Oct 30 13:06:01.606998 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.606923    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f163e2e9-ace0-44b6-962c-19b61fe31c95-kube-api-access-htnsz\" (UniqueName: \"kubernetes.io/projected/f163e2e9-ace0-44b6-962c-19b61fe31c95-kube-api-access-htnsz\") pod \"f163e2e9-ace0-44b6-962c-19b61fe31c95\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") "
Oct 30 13:06:01.606998 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.606971    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-credential-socket\") pod \"f163e2e9-ace0-44b6-962c-19b61fe31c95\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") "
Oct 30 13:06:01.607169 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.607001    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-envoy\") pod \"f163e2e9-ace0-44b6-962c-19b61fe31c95\" (UID: \"f163e2e9-ace0-44b6-962c-19b61fe31c95\") "
Oct 30 13:06:01.615192 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.611372    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-token" pod "f163e2e9-ace0-44b6-962c-19b61fe31c95" (UID: "f163e2e9-ace0-44b6-962c-19b61fe31c95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:01.615192 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.611704    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-credential-socket" pod "f163e2e9-ace0-44b6-962c-19b61fe31c95" (UID: "f163e2e9-ace0-44b6-962c-19b61fe31c95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:01.615192 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.611994    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f163e2e9-ace0-44b6-962c-19b61fe31c95-istiod-ca-cert" pod "f163e2e9-ace0-44b6-962c-19b61fe31c95" (UID: "f163e2e9-ace0-44b6-962c-19b61fe31c95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:01.615192 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.612083    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-workload-certs" pod "f163e2e9-ace0-44b6-962c-19b61fe31c95" (UID: "f163e2e9-ace0-44b6-962c-19b61fe31c95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:01.615192 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.612131    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-data" pod "f163e2e9-ace0-44b6-962c-19b61fe31c95" (UID: "f163e2e9-ace0-44b6-962c-19b61fe31c95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:01.615192 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.612178    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-workload-socket" pod "f163e2e9-ace0-44b6-962c-19b61fe31c95" (UID: "f163e2e9-ace0-44b6-962c-19b61fe31c95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:01.615192 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.615089    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-envoy" pod "f163e2e9-ace0-44b6-962c-19b61fe31c95" (UID: "f163e2e9-ace0-44b6-962c-19b61fe31c95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:01.622957 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.622920    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f163e2e9-ace0-44b6-962c-19b61fe31c95-kube-api-access-htnsz" pod "f163e2e9-ace0-44b6-962c-19b61fe31c95" (UID: "f163e2e9-ace0-44b6-962c-19b61fe31c95"). InnerVolumeSpecName "kube-api-access-htnsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:01.625611 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.625576    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-podinfo" pod "f163e2e9-ace0-44b6-962c-19b61fe31c95" (UID: "f163e2e9-ace0-44b6-962c-19b61fe31c95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:06:01.709033 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.708997    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f163e2e9-ace0-44b6-962c-19b61fe31c95-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:01.709033 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.709032    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:01.709268 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.709047    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:01.709268 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.709060    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:01.709268 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.709072    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:01.709268 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.709084    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:01.709268 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.709098    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htnsz\" (UniqueName: \"kubernetes.io/projected/f163e2e9-ace0-44b6-962c-19b61fe31c95-kube-api-access-htnsz\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:01.709268 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.709112    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:01.709268 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.709126    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f163e2e9-ace0-44b6-962c-19b61fe31c95-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:01.913532 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.913452    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"]
Oct 30 13:06:01.914485 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.914464    5837 cpu_manager.go:414] "containerMap: removing container" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerName="istio-proxy"
Oct 30 13:06:01.914485 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.914484    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerName="istio-proxy"
Oct 30 13:06:01.914672 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.914563    5837 cpu_manager.go:414] "containerMap: removing container" podUID="797bf104-289c-4c41-97b8-9413b316dba2" containerName="revoke-keys"
Oct 30 13:06:01.914672 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.914571    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="797bf104-289c-4c41-97b8-9413b316dba2" containerName="revoke-keys"
Oct 30 13:06:01.914672 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.914596    5837 cpu_manager.go:414] "containerMap: removing container" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerName="istio-proxy"
Oct 30 13:06:01.914672 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.914605    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerName="istio-proxy"
Oct 30 13:06:01.914800 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.914749    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerName="istio-proxy"
Oct 30 13:06:01.914800 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.914772    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="797bf104-289c-4c41-97b8-9413b316dba2" containerName="revoke-keys"
Oct 30 13:06:01.920495 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.920468    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"
Oct 30 13:06:01.948528 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.948486    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"]
Oct 30 13:06:01.951817 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.951781    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:06:01.952006 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.951859    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:06:01.955724 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.955695    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:06:01.955724 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:01.955701    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:06:02.013171 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.013129    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb5cc095-af95-4463-9e95-000901d19a91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f06d9ee-vz6t5\" (UID: \"cb5cc095-af95-4463-9e95-000901d19a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"
Oct 30 13:06:02.013356 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.013294    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxp6q\" (UniqueName: \"kubernetes.io/projected/cb5cc095-af95-4463-9e95-000901d19a91-kube-api-access-nxp6q\") pod \"maas-api-revoke-sub-1f06d9ee-vz6t5\" (UID: \"cb5cc095-af95-4463-9e95-000901d19a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"
Oct 30 13:06:02.013537 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.013518    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb5cc095-af95-4463-9e95-000901d19a91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f06d9ee-vz6t5\" (UID: \"cb5cc095-af95-4463-9e95-000901d19a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"
Oct 30 13:06:02.042349 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.042304    5837 generic.go:356] "Generic (PLEG): container finished" podID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerID="06a664ce2615e78d052b6912976f70f30db9f34c5b2cbf3c006260482514bec4" exitCode=0
Oct 30 13:06:02.042549 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.042348    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" event={"ID":"f163e2e9-ace0-44b6-962c-19b61fe31c95","Type":"ContainerDied","Data":"06a664ce2615e78d052b6912976f70f30db9f34c5b2cbf3c006260482514bec4"}
Oct 30 13:06:02.042549 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.042375    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"
Oct 30 13:06:02.042549 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.042427    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74" event={"ID":"f163e2e9-ace0-44b6-962c-19b61fe31c95","Type":"ContainerDied","Data":"ce708932df8c1d7501e2313ffad59c488387ff06f7379341b10decb9aefbeb01"}
Oct 30 13:06:02.042549 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.042459    5837 scope.go:122] "RemoveContainer" containerID="06a664ce2615e78d052b6912976f70f30db9f34c5b2cbf3c006260482514bec4"
Oct 30 13:06:02.056026 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.056006    5837 scope.go:122] "RemoveContainer" containerID="3f11e4b460fb43ff4546577e286bbb4010d22453f87ba60ea169e58a6e0f7f48"
Oct 30 13:06:02.068950 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.068869    5837 scope.go:122] "RemoveContainer" containerID="06a664ce2615e78d052b6912976f70f30db9f34c5b2cbf3c006260482514bec4"
Oct 30 13:06:02.069205 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:02.069176    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06a664ce2615e78d052b6912976f70f30db9f34c5b2cbf3c006260482514bec4\": container with ID starting with 06a664ce2615e78d052b6912976f70f30db9f34c5b2cbf3c006260482514bec4 not found: ID does not exist" containerID="06a664ce2615e78d052b6912976f70f30db9f34c5b2cbf3c006260482514bec4"
Oct 30 13:06:02.069316 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.069212    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a664ce2615e78d052b6912976f70f30db9f34c5b2cbf3c006260482514bec4"} err="failed to get container status \"06a664ce2615e78d052b6912976f70f30db9f34c5b2cbf3c006260482514bec4\": rpc error: code = NotFound desc = could not find container \"06a664ce2615e78d052b6912976f70f30db9f34c5b2cbf3c006260482514bec4\": container with ID starting with 06a664ce2615e78d052b6912976f70f30db9f34c5b2cbf3c006260482514bec4 not found: ID does not exist"
Oct 30 13:06:02.069316 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.069229    5837 scope.go:122] "RemoveContainer" containerID="3f11e4b460fb43ff4546577e286bbb4010d22453f87ba60ea169e58a6e0f7f48"
Oct 30 13:06:02.069567 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:02.069550    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f11e4b460fb43ff4546577e286bbb4010d22453f87ba60ea169e58a6e0f7f48\": container with ID starting with 3f11e4b460fb43ff4546577e286bbb4010d22453f87ba60ea169e58a6e0f7f48 not found: ID does not exist" containerID="3f11e4b460fb43ff4546577e286bbb4010d22453f87ba60ea169e58a6e0f7f48"
Oct 30 13:06:02.069633 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.069574    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f11e4b460fb43ff4546577e286bbb4010d22453f87ba60ea169e58a6e0f7f48"} err="failed to get container status \"3f11e4b460fb43ff4546577e286bbb4010d22453f87ba60ea169e58a6e0f7f48\": rpc error: code = NotFound desc = could not find container \"3f11e4b460fb43ff4546577e286bbb4010d22453f87ba60ea169e58a6e0f7f48\": container with ID starting with 3f11e4b460fb43ff4546577e286bbb4010d22453f87ba60ea169e58a6e0f7f48 not found: ID does not exist"
Oct 30 13:06:02.098297 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.098266    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"]
Oct 30 13:06:02.108454 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.108426    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-1f91c1-openshift-default-688c4d5857-qzj74"]
Oct 30 13:06:02.115309 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.115276    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb5cc095-af95-4463-9e95-000901d19a91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f06d9ee-vz6t5\" (UID: \"cb5cc095-af95-4463-9e95-000901d19a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"
Oct 30 13:06:02.115464 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.115436    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb5cc095-af95-4463-9e95-000901d19a91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f06d9ee-vz6t5\" (UID: \"cb5cc095-af95-4463-9e95-000901d19a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"
Oct 30 13:06:02.115580 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.115556    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxp6q\" (UniqueName: \"kubernetes.io/projected/cb5cc095-af95-4463-9e95-000901d19a91-kube-api-access-nxp6q\") pod \"maas-api-revoke-sub-1f06d9ee-vz6t5\" (UID: \"cb5cc095-af95-4463-9e95-000901d19a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"
Oct 30 13:06:02.116164 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.116143    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb5cc095-af95-4463-9e95-000901d19a91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f06d9ee-vz6t5\" (UID: \"cb5cc095-af95-4463-9e95-000901d19a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"
Oct 30 13:06:02.118043 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.118022    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb5cc095-af95-4463-9e95-000901d19a91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f06d9ee-vz6t5\" (UID: \"cb5cc095-af95-4463-9e95-000901d19a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"
Oct 30 13:06:02.135574 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.135540    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxp6q\" (UniqueName: \"kubernetes.io/projected/cb5cc095-af95-4463-9e95-000901d19a91-kube-api-access-nxp6q\") pod \"maas-api-revoke-sub-1f06d9ee-vz6t5\" (UID: \"cb5cc095-af95-4463-9e95-000901d19a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"
Oct 30 13:06:02.236696 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.236659    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"
Oct 30 13:06:02.556577 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:02.556547    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"]
Oct 30 13:06:02.556935 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:06:02.556792    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5cc095_af95_4463_9e95_000901d19a91.slice/crio-512e01c59e62e7b794bbfa19df777d7a1bcaecd2374adf1ecf857312c3d5cc3c WatchSource:0}: Error finding container 512e01c59e62e7b794bbfa19df777d7a1bcaecd2374adf1ecf857312c3d5cc3c: Status 404 returned error can't find the container with id 512e01c59e62e7b794bbfa19df777d7a1bcaecd2374adf1ecf857312c3d5cc3c
Oct 30 13:06:03.053972 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:03.053932    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5" event={"ID":"cb5cc095-af95-4463-9e95-000901d19a91","Type":"ContainerStarted","Data":"512e01c59e62e7b794bbfa19df777d7a1bcaecd2374adf1ecf857312c3d5cc3c"}
Oct 30 13:06:03.607968 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:03.607934    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" path="/var/lib/kubelet/pods/f163e2e9-ace0-44b6-962c-19b61fe31c95/volumes"
Oct 30 13:06:04.065237 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:04.065203    5837 generic.go:356] "Generic (PLEG): container finished" podID="cb5cc095-af95-4463-9e95-000901d19a91" containerID="e87a4e4f44803e5afc6d0535d5acde1ac7e252c6559433f4052a1d91953b2bf3" exitCode=0
Oct 30 13:06:04.065555 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:04.065295    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5" event={"ID":"cb5cc095-af95-4463-9e95-000901d19a91","Type":"ContainerDied","Data":"e87a4e4f44803e5afc6d0535d5acde1ac7e252c6559433f4052a1d91953b2bf3"}
Oct 30 13:06:05.074308 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.074272    5837 generic.go:356] "Generic (PLEG): container finished" podID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerID="90ed49c8e55f0938be4d90747c666e3f6cdd7c4f59c568ab77f7e87bf7efd192" exitCode=0
Oct 30 13:06:05.074821 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.074508    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs" event={"ID":"12c0a91a-fa80-4a17-9278-6bc43b6b9847","Type":"ContainerDied","Data":"90ed49c8e55f0938be4d90747c666e3f6cdd7c4f59c568ab77f7e87bf7efd192"}
Oct 30 13:06:05.075164 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.075098    5837 scope.go:122] "RemoveContainer" containerID="90ed49c8e55f0938be4d90747c666e3f6cdd7c4f59c568ab77f7e87bf7efd192"
Oct 30 13:06:05.076610 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.076582    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:06:05.076742 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.076649    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:06:05.076742 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.076681    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:06:05.182916 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.182886    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"]
Oct 30 13:06:05.215451 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.215414    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"]
Oct 30 13:06:05.263333 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.263300    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"
Oct 30 13:06:05.389240 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.389152    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb5cc095-af95-4463-9e95-000901d19a91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb5cc095-af95-4463-9e95-000901d19a91-maas-api-openshift-ca\") pod \"cb5cc095-af95-4463-9e95-000901d19a91\" (UID: \"cb5cc095-af95-4463-9e95-000901d19a91\") "
Oct 30 13:06:05.389434 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.389297    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb5cc095-af95-4463-9e95-000901d19a91-kube-api-access-nxp6q\" (UniqueName: \"kubernetes.io/projected/cb5cc095-af95-4463-9e95-000901d19a91-kube-api-access-nxp6q\") pod \"cb5cc095-af95-4463-9e95-000901d19a91\" (UID: \"cb5cc095-af95-4463-9e95-000901d19a91\") "
Oct 30 13:06:05.389540 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.389523    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb5cc095-af95-4463-9e95-000901d19a91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb5cc095-af95-4463-9e95-000901d19a91-maas-api-xks-ca\") pod \"cb5cc095-af95-4463-9e95-000901d19a91\" (UID: \"cb5cc095-af95-4463-9e95-000901d19a91\") "
Oct 30 13:06:05.389680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.389522    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5cc095-af95-4463-9e95-000901d19a91-maas-api-openshift-ca" pod "cb5cc095-af95-4463-9e95-000901d19a91" (UID: "cb5cc095-af95-4463-9e95-000901d19a91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:05.390149 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.390128    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb5cc095-af95-4463-9e95-000901d19a91-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:05.391692 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.391660    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5cc095-af95-4463-9e95-000901d19a91-kube-api-access-nxp6q" pod "cb5cc095-af95-4463-9e95-000901d19a91" (UID: "cb5cc095-af95-4463-9e95-000901d19a91"). InnerVolumeSpecName "kube-api-access-nxp6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:05.391831 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.391811    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5cc095-af95-4463-9e95-000901d19a91-maas-api-xks-ca" pod "cb5cc095-af95-4463-9e95-000901d19a91" (UID: "cb5cc095-af95-4463-9e95-000901d19a91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:05.491717 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.491673    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxp6q\" (UniqueName: \"kubernetes.io/projected/cb5cc095-af95-4463-9e95-000901d19a91-kube-api-access-nxp6q\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:05.491717 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:05.491718    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb5cc095-af95-4463-9e95-000901d19a91-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:06.084583 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:06.084552    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"
Oct 30 13:06:06.085089 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:06.084545    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5" event={"ID":"cb5cc095-af95-4463-9e95-000901d19a91","Type":"ContainerDied","Data":"512e01c59e62e7b794bbfa19df777d7a1bcaecd2374adf1ecf857312c3d5cc3c"}
Oct 30 13:06:06.085089 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:06.084698    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="512e01c59e62e7b794bbfa19df777d7a1bcaecd2374adf1ecf857312c3d5cc3c"
Oct 30 13:06:06.087009 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:06.086983    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs" event={"ID":"12c0a91a-fa80-4a17-9278-6bc43b6b9847","Type":"ContainerStarted","Data":"1e137026135c14ff1fa6c5a001f792310a8472789567563ac88b32a81ebb1362"}
Oct 30 13:06:06.544800 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:06.544760    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:06:06.544999 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:06.544975    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 13:06:06.545050 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:06.545021    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs" podUID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 13:06:07.031265 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.031228    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"]
Oct 30 13:06:07.040426 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.034767    5837 cpu_manager.go:414] "containerMap: removing container" podUID="cb5cc095-af95-4463-9e95-000901d19a91" containerName="revoke-keys"
Oct 30 13:06:07.040707 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.040671    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb5cc095-af95-4463-9e95-000901d19a91" containerName="revoke-keys"
Oct 30 13:06:07.041207 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.041160    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb5cc095-af95-4463-9e95-000901d19a91" containerName="revoke-keys"
Oct 30 13:06:07.041441 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.041418    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f163e2e9-ace0-44b6-962c-19b61fe31c95" containerName="istio-proxy"
Oct 30 13:06:07.048467 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.048441    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.070957 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.069644    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-4cb0fe-openshift-default-dockercfg-4ftfs\""
Oct 30 13:06:07.071136 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.071002    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"]
Oct 30 13:06:07.138428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.132161    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-workload-certs\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.138428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.132240    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-workload-socket\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.138428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.132276    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-istio-data\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.138428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.132341    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ff36816-2f3d-4007-b635-2a79923951f1-istiod-ca-cert\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.138428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.132428    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ff36816-2f3d-4007-b635-2a79923951f1-istio-podinfo\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.138428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.132604    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-istio-envoy\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.138428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.132632    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhcr9\" (UniqueName: \"kubernetes.io/projected/4ff36816-2f3d-4007-b635-2a79923951f1-kube-api-access-hhcr9\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.138428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.132667    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-credential-socket\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.138428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.132700    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ff36816-2f3d-4007-b635-2a79923951f1-istio-token\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.286465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.270203    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ff36816-2f3d-4007-b635-2a79923951f1-istio-podinfo\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.286465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.270428    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-istio-envoy\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.286465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.270463    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhcr9\" (UniqueName: \"kubernetes.io/projected/4ff36816-2f3d-4007-b635-2a79923951f1-kube-api-access-hhcr9\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.286465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.270503    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-credential-socket\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.286465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.270568    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ff36816-2f3d-4007-b635-2a79923951f1-istio-token\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.286465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.270649    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-workload-certs\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.286465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.270731    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-workload-socket\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.286465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.270762    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-istio-data\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.286465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.270855    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ff36816-2f3d-4007-b635-2a79923951f1-istiod-ca-cert\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.286465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.272042    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-workload-certs\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.286465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.272772    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-credential-socket\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.286465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.274385    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ff36816-2f3d-4007-b635-2a79923951f1-istiod-ca-cert\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.286465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.274642    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-workload-socket\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.286465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.274805    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-istio-data\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.428420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.385762    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-istio-envoy\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.433192 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.433156    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ff36816-2f3d-4007-b635-2a79923951f1-istio-podinfo\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.479519 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.477239    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhcr9\" (UniqueName: \"kubernetes.io/projected/4ff36816-2f3d-4007-b635-2a79923951f1-kube-api-access-hhcr9\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.487210 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.487148    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ff36816-2f3d-4007-b635-2a79923951f1-istio-token\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:07.544031 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.543951    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 13:06:07.544031 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.544009    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs" podUID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 13:06:07.557734 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:07.557702    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice/crio-eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:06:07.672953 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.672913    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"]
Oct 30 13:06:07.678274 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.678238    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f73c9f-sh729"]
Oct 30 13:06:07.708424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:07.708021    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:08.114384 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:08.114295    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"]
Oct 30 13:06:08.119593 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:06:08.119564    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff36816_2f3d_4007_b635_2a79923951f1.slice/crio-fd2ef9555f65c7678708077b779b99e73e0d6f509c766ff558d5b7bda69bcb83 WatchSource:0}: Error finding container fd2ef9555f65c7678708077b779b99e73e0d6f509c766ff558d5b7bda69bcb83: Status 404 returned error can't find the container with id fd2ef9555f65c7678708077b779b99e73e0d6f509c766ff558d5b7bda69bcb83
Oct 30 13:06:08.122461 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:08.122380    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:06:08.122570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:08.122511    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:06:08.122655 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:08.122571    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:06:08.200559 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:08.200467    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b" event={"ID":"4ff36816-2f3d-4007-b635-2a79923951f1","Type":"ContainerStarted","Data":"fd2ef9555f65c7678708077b779b99e73e0d6f509c766ff558d5b7bda69bcb83"}
Oct 30 13:06:08.544177 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:08.544143    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 13:06:08.544355 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:08.544202    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs" podUID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 13:06:08.571109 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:08.571065    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice/crio-eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:06:08.571265 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:08.571073    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice/crio-eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:06:09.241104 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:09.241062    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b" event={"ID":"4ff36816-2f3d-4007-b635-2a79923951f1","Type":"ContainerStarted","Data":"be0bf6100b94a83d02cc48323450ac52747455527d0d141b714a8144aa0e3211"}
Oct 30 13:06:09.320420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:09.319810    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b" podStartSLOduration=3.31978516 podStartE2EDuration="3.31978516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:09.299184751 +0000 UTC m=+1757.823468971" watchObservedRunningTime="2026-10-30 13:06:09.31978516 +0000 UTC m=+1757.844069380"
Oct 30 13:06:09.544524 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:09.544422    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 13:06:09.544524 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:09.544500    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs" podUID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 13:06:09.642581 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:09.642542    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="797bf104-289c-4c41-97b8-9413b316dba2" path="/var/lib/kubelet/pods/797bf104-289c-4c41-97b8-9413b316dba2/volumes"
Oct 30 13:06:09.712453 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:09.711576    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:09.712453 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:09.711711    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:06:09.712453 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:09.711752    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b" podUID="4ff36816-2f3d-4007-b635-2a79923951f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 13:06:10.183192 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.183143    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th" podUID="bbb1a7e8-4931-4264-bc9d-194832b35e35" containerName="payload-processing" containerID="cri-o://1e685b86b9d37e52173e6e1beda62d1985a06d025e949514a627a4889804c212" gracePeriod=25
Oct 30 13:06:10.223482 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.217853    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq" podUID="f5109255-b08f-4f6e-a83d-325b173f11be" containerName="payload-pre-processing" containerID="cri-o://9208d448a8b595d53369031b640abe967ef2e998f844bb8904d5c6d5660ced02" gracePeriod=25
Oct 30 13:06:10.546275 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.546005    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 13:06:10.546275 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.546083    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs" podUID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 13:06:10.649271 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.649241    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"
Oct 30 13:06:10.748847 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.748484    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb1a7e8-4931-4264-bc9d-194832b35e35-kube-api-access-hhht7\" (UniqueName: \"kubernetes.io/projected/bbb1a7e8-4931-4264-bc9d-194832b35e35-kube-api-access-hhht7\") pod \"bbb1a7e8-4931-4264-bc9d-194832b35e35\" (UID: \"bbb1a7e8-4931-4264-bc9d-194832b35e35\") "
Oct 30 13:06:10.748847 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.748791    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbb1a7e8-4931-4264-bc9d-194832b35e35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb1a7e8-4931-4264-bc9d-194832b35e35-plugins-config-volume\") pod \"bbb1a7e8-4931-4264-bc9d-194832b35e35\" (UID: \"bbb1a7e8-4931-4264-bc9d-194832b35e35\") "
Oct 30 13:06:10.748847 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.748796    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:06:10.749240 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.749192    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb1a7e8-4931-4264-bc9d-194832b35e35-plugins-config-volume" pod "bbb1a7e8-4931-4264-bc9d-194832b35e35" (UID: "bbb1a7e8-4931-4264-bc9d-194832b35e35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:10.749311 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.749283    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b" podUID="4ff36816-2f3d-4007-b635-2a79923951f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 13:06:10.750142 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.750111    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb1a7e8-4931-4264-bc9d-194832b35e35-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:10.762261 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.762236    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb1a7e8-4931-4264-bc9d-194832b35e35-kube-api-access-hhht7" pod "bbb1a7e8-4931-4264-bc9d-194832b35e35" (UID: "bbb1a7e8-4931-4264-bc9d-194832b35e35"). InnerVolumeSpecName "kube-api-access-hhht7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:10.846431 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.843431    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"
Oct 30 13:06:10.856139 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.856108    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhht7\" (UniqueName: \"kubernetes.io/projected/bbb1a7e8-4931-4264-bc9d-194832b35e35-kube-api-access-hhht7\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:10.860619 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.860595    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"]
Oct 30 13:06:10.861427 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.861410    5837 cpu_manager.go:414] "containerMap: removing container" podUID="bbb1a7e8-4931-4264-bc9d-194832b35e35" containerName="payload-processing"
Oct 30 13:06:10.861535 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.861523    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb1a7e8-4931-4264-bc9d-194832b35e35" containerName="payload-processing"
Oct 30 13:06:10.861681 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.861669    5837 cpu_manager.go:414] "containerMap: removing container" podUID="f5109255-b08f-4f6e-a83d-325b173f11be" containerName="payload-pre-processing"
Oct 30 13:06:10.861779 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.861766    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5109255-b08f-4f6e-a83d-325b173f11be" containerName="payload-pre-processing"
Oct 30 13:06:10.862100 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.862084    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5109255-b08f-4f6e-a83d-325b173f11be" containerName="payload-pre-processing"
Oct 30 13:06:10.862231 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.862219    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb1a7e8-4931-4264-bc9d-194832b35e35" containerName="payload-processing"
Oct 30 13:06:10.865952 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.865930    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"
Oct 30 13:06:10.867481 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.867450    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"]
Oct 30 13:06:10.882303 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.881293    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:06:10.882303 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.881437    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:06:10.882303 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.881892    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:06:10.882303 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.882019    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:06:10.957469 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.957433    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5109255-b08f-4f6e-a83d-325b173f11be-kube-api-access-pjxkg\" (UniqueName: \"kubernetes.io/projected/f5109255-b08f-4f6e-a83d-325b173f11be-kube-api-access-pjxkg\") pod \"f5109255-b08f-4f6e-a83d-325b173f11be\" (UID: \"f5109255-b08f-4f6e-a83d-325b173f11be\") "
Oct 30 13:06:10.957722 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.957699    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5109255-b08f-4f6e-a83d-325b173f11be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5109255-b08f-4f6e-a83d-325b173f11be-plugins-config-volume\") pod \"f5109255-b08f-4f6e-a83d-325b173f11be\" (UID: \"f5109255-b08f-4f6e-a83d-325b173f11be\") "
Oct 30 13:06:10.958099 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.958046    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5109255-b08f-4f6e-a83d-325b173f11be-plugins-config-volume" pod "f5109255-b08f-4f6e-a83d-325b173f11be" (UID: "f5109255-b08f-4f6e-a83d-325b173f11be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:10.958411 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.958368    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5109255-b08f-4f6e-a83d-325b173f11be-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:10.960208 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:10.960179    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5109255-b08f-4f6e-a83d-325b173f11be-kube-api-access-pjxkg" pod "f5109255-b08f-4f6e-a83d-325b173f11be" (UID: "f5109255-b08f-4f6e-a83d-325b173f11be"). InnerVolumeSpecName "kube-api-access-pjxkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:11.060047 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.059959    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j\" (UID: \"0ef5ae74-ccbc-4e61-b92c-72d35130cb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"
Oct 30 13:06:11.060189 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.060071    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbpxl\" (UniqueName: \"kubernetes.io/projected/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-kube-api-access-kbpxl\") pod \"maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j\" (UID: \"0ef5ae74-ccbc-4e61-b92c-72d35130cb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"
Oct 30 13:06:11.060246 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.060189    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j\" (UID: \"0ef5ae74-ccbc-4e61-b92c-72d35130cb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"
Oct 30 13:06:11.060298 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.060285    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjxkg\" (UniqueName: \"kubernetes.io/projected/f5109255-b08f-4f6e-a83d-325b173f11be-kube-api-access-pjxkg\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:11.161381 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.161333    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j\" (UID: \"0ef5ae74-ccbc-4e61-b92c-72d35130cb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"
Oct 30 13:06:11.161603 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.161459    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbpxl\" (UniqueName: \"kubernetes.io/projected/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-kube-api-access-kbpxl\") pod \"maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j\" (UID: \"0ef5ae74-ccbc-4e61-b92c-72d35130cb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"
Oct 30 13:06:11.161699 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.161617    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j\" (UID: \"0ef5ae74-ccbc-4e61-b92c-72d35130cb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"
Oct 30 13:06:11.162446 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.162301    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j\" (UID: \"0ef5ae74-ccbc-4e61-b92c-72d35130cb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"
Oct 30 13:06:11.164199 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.164176    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j\" (UID: \"0ef5ae74-ccbc-4e61-b92c-72d35130cb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"
Oct 30 13:06:11.180826 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.180799    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbpxl\" (UniqueName: \"kubernetes.io/projected/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-kube-api-access-kbpxl\") pod \"maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j\" (UID: \"0ef5ae74-ccbc-4e61-b92c-72d35130cb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"
Oct 30 13:06:11.195418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.187593    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"
Oct 30 13:06:11.261089 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.261062    5837 generic.go:356] "Generic (PLEG): container finished" podID="f5109255-b08f-4f6e-a83d-325b173f11be" containerID="9208d448a8b595d53369031b640abe967ef2e998f844bb8904d5c6d5660ced02" exitCode=0
Oct 30 13:06:11.261230 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.261216    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq" event={"ID":"f5109255-b08f-4f6e-a83d-325b173f11be","Type":"ContainerDied","Data":"9208d448a8b595d53369031b640abe967ef2e998f844bb8904d5c6d5660ced02"}
Oct 30 13:06:11.261283 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.261255    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq" event={"ID":"f5109255-b08f-4f6e-a83d-325b173f11be","Type":"ContainerDied","Data":"78a2c37aaa5cd10e5c828a6dc7887af42d7de7d5f49b98d9be193a6eda394d43"}
Oct 30 13:06:11.261323 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.261275    5837 scope.go:122] "RemoveContainer" containerID="9208d448a8b595d53369031b640abe967ef2e998f844bb8904d5c6d5660ced02"
Oct 30 13:06:11.261466 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.261455    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"
Oct 30 13:06:11.268759 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.268732    5837 generic.go:356] "Generic (PLEG): container finished" podID="bbb1a7e8-4931-4264-bc9d-194832b35e35" containerID="1e685b86b9d37e52173e6e1beda62d1985a06d025e949514a627a4889804c212" exitCode=0
Oct 30 13:06:11.269510 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.269489    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"
Oct 30 13:06:11.271599 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.271573    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th" event={"ID":"bbb1a7e8-4931-4264-bc9d-194832b35e35","Type":"ContainerDied","Data":"1e685b86b9d37e52173e6e1beda62d1985a06d025e949514a627a4889804c212"}
Oct 30 13:06:11.271691 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.271630    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th" event={"ID":"bbb1a7e8-4931-4264-bc9d-194832b35e35","Type":"ContainerDied","Data":"fa36e4cac95fcc32fb79e06f28715704c8808db9608ed7e1d41c2209a73ca5a1"}
Oct 30 13:06:11.282826 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.282807    5837 scope.go:122] "RemoveContainer" containerID="9208d448a8b595d53369031b640abe967ef2e998f844bb8904d5c6d5660ced02"
Oct 30 13:06:11.283116 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:11.283095    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9208d448a8b595d53369031b640abe967ef2e998f844bb8904d5c6d5660ced02\": container with ID starting with 9208d448a8b595d53369031b640abe967ef2e998f844bb8904d5c6d5660ced02 not found: ID does not exist" containerID="9208d448a8b595d53369031b640abe967ef2e998f844bb8904d5c6d5660ced02"
Oct 30 13:06:11.283184 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.283127    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9208d448a8b595d53369031b640abe967ef2e998f844bb8904d5c6d5660ced02"} err="failed to get container status \"9208d448a8b595d53369031b640abe967ef2e998f844bb8904d5c6d5660ced02\": rpc error: code = NotFound desc = could not find container \"9208d448a8b595d53369031b640abe967ef2e998f844bb8904d5c6d5660ced02\": container with ID starting with 9208d448a8b595d53369031b640abe967ef2e998f844bb8904d5c6d5660ced02 not found: ID does not exist"
Oct 30 13:06:11.283184 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.283150    5837 scope.go:122] "RemoveContainer" containerID="1e685b86b9d37e52173e6e1beda62d1985a06d025e949514a627a4889804c212"
Oct 30 13:06:11.350136 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.347931    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"]
Oct 30 13:06:11.356031 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.352167    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f73c9f-6d96bf5f67-2k9xq"]
Oct 30 13:06:11.365136 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.365108    5837 scope.go:122] "RemoveContainer" containerID="1e685b86b9d37e52173e6e1beda62d1985a06d025e949514a627a4889804c212"
Oct 30 13:06:11.367483 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:11.367444    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e685b86b9d37e52173e6e1beda62d1985a06d025e949514a627a4889804c212\": container with ID starting with 1e685b86b9d37e52173e6e1beda62d1985a06d025e949514a627a4889804c212 not found: ID does not exist" containerID="1e685b86b9d37e52173e6e1beda62d1985a06d025e949514a627a4889804c212"
Oct 30 13:06:11.367667 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.367495    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e685b86b9d37e52173e6e1beda62d1985a06d025e949514a627a4889804c212"} err="failed to get container status \"1e685b86b9d37e52173e6e1beda62d1985a06d025e949514a627a4889804c212\": rpc error: code = NotFound desc = could not find container \"1e685b86b9d37e52173e6e1beda62d1985a06d025e949514a627a4889804c212\": container with ID starting with 1e685b86b9d37e52173e6e1beda62d1985a06d025e949514a627a4889804c212 not found: ID does not exist"
Oct 30 13:06:11.377834 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.377796    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"]
Oct 30 13:06:11.382576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.382556    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f73c9f-79dcb4b588-jq6th"]
Oct 30 13:06:11.544756 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.544722    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 13:06:11.544906 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.544768    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs" podUID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 13:06:11.571287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.571261    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"]
Oct 30 13:06:11.572272 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:06:11.572245    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef5ae74_ccbc_4e61_b92c_72d35130cb10.slice/crio-07af8ef5bdbd68b18a81c7393bbea5f2b23635e9044f063983d3d0dd66813eca WatchSource:0}: Error finding container 07af8ef5bdbd68b18a81c7393bbea5f2b23635e9044f063983d3d0dd66813eca: Status 404 returned error can't find the container with id 07af8ef5bdbd68b18a81c7393bbea5f2b23635e9044f063983d3d0dd66813eca
Oct 30 13:06:11.642898 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.642808    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbb1a7e8-4931-4264-bc9d-194832b35e35" path="/var/lib/kubelet/pods/bbb1a7e8-4931-4264-bc9d-194832b35e35/volumes"
Oct 30 13:06:11.643539 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.643504    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5109255-b08f-4f6e-a83d-325b173f11be" path="/var/lib/kubelet/pods/f5109255-b08f-4f6e-a83d-325b173f11be/volumes"
Oct 30 13:06:11.710897 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.710858    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:06:11.733992 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:11.733946    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b" podUID="4ff36816-2f3d-4007-b635-2a79923951f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 13:06:12.279122 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:12.279081    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j" event={"ID":"0ef5ae74-ccbc-4e61-b92c-72d35130cb10","Type":"ContainerStarted","Data":"45318c9901b081287a0ad925698afd7f46cf8ef28ae44011492f8ff8fe63a75f"}
Oct 30 13:06:12.279298 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:12.279149    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j" event={"ID":"0ef5ae74-ccbc-4e61-b92c-72d35130cb10","Type":"ContainerStarted","Data":"07af8ef5bdbd68b18a81c7393bbea5f2b23635e9044f063983d3d0dd66813eca"}
Oct 30 13:06:12.329186 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:12.329130    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j" podStartSLOduration=2.329115864 podStartE2EDuration="2.329115864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:12.326524493 +0000 UTC m=+1760.850808714" watchObservedRunningTime="2026-10-30 13:06:12.329115864 +0000 UTC m=+1760.853400082"
Oct 30 13:06:12.548788 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:12.548708    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:06:12.549043 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:12.549024    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:06:12.549781 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:12.549759    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:06:12.715449 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:12.715416    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:06:12.716027 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:12.715993    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b" podUID="4ff36816-2f3d-4007-b635-2a79923951f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 13:06:13.312638 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:13.312601    5837 generic.go:356] "Generic (PLEG): container finished" podID="0ef5ae74-ccbc-4e61-b92c-72d35130cb10" containerID="45318c9901b081287a0ad925698afd7f46cf8ef28ae44011492f8ff8fe63a75f" exitCode=0
Oct 30 13:06:13.312827 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:13.312686    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j" event={"ID":"0ef5ae74-ccbc-4e61-b92c-72d35130cb10","Type":"ContainerDied","Data":"45318c9901b081287a0ad925698afd7f46cf8ef28ae44011492f8ff8fe63a75f"}
Oct 30 13:06:13.709074 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:13.708981    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:06:13.709074 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:13.709042    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b" podUID="4ff36816-2f3d-4007-b635-2a79923951f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 13:06:14.551111 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:14.551079    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"
Oct 30 13:06:14.636683 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:14.636556    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"]
Oct 30 13:06:14.637010 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:14.636975    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy" containerID="cri-o://4f5bed7edfad0ee5957c1eb14b931952d5ca8db0c6cf7d61dcd9a782ce510ced" gracePeriod=30
Oct 30 13:06:14.651061 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:14.651021    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-maas-api-xks-ca\") pod \"0ef5ae74-ccbc-4e61-b92c-72d35130cb10\" (UID: \"0ef5ae74-ccbc-4e61-b92c-72d35130cb10\") "
Oct 30 13:06:14.651350 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:14.651145    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-kube-api-access-kbpxl\" (UniqueName: \"kubernetes.io/projected/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-kube-api-access-kbpxl\") pod \"0ef5ae74-ccbc-4e61-b92c-72d35130cb10\" (UID: \"0ef5ae74-ccbc-4e61-b92c-72d35130cb10\") "
Oct 30 13:06:14.651350 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:14.651305    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-maas-api-openshift-ca\") pod \"0ef5ae74-ccbc-4e61-b92c-72d35130cb10\" (UID: \"0ef5ae74-ccbc-4e61-b92c-72d35130cb10\") "
Oct 30 13:06:14.652282 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:14.652240    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-maas-api-openshift-ca" pod "0ef5ae74-ccbc-4e61-b92c-72d35130cb10" (UID: "0ef5ae74-ccbc-4e61-b92c-72d35130cb10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:14.655526 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:14.655472    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-kube-api-access-kbpxl" pod "0ef5ae74-ccbc-4e61-b92c-72d35130cb10" (UID: "0ef5ae74-ccbc-4e61-b92c-72d35130cb10"). InnerVolumeSpecName "kube-api-access-kbpxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:14.724773 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:14.723827    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:06:14.724773 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:14.723886    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b" podUID="4ff36816-2f3d-4007-b635-2a79923951f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 13:06:14.763432 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:14.762966    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:14.763432 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:14.762998    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbpxl\" (UniqueName: \"kubernetes.io/projected/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-kube-api-access-kbpxl\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:14.769756 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:14.766811    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-maas-api-xks-ca" pod "0ef5ae74-ccbc-4e61-b92c-72d35130cb10" (UID: "0ef5ae74-ccbc-4e61-b92c-72d35130cb10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:14.874810 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:14.874759    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef5ae74-ccbc-4e61-b92c-72d35130cb10-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:15.097466 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:15.097426    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:06:15.147677 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.147639    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"]
Oct 30 13:06:15.148762 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.148739    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0ef5ae74-ccbc-4e61-b92c-72d35130cb10" containerName="revoke-keys"
Oct 30 13:06:15.148815 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.148767    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ef5ae74-ccbc-4e61-b92c-72d35130cb10" containerName="revoke-keys"
Oct 30 13:06:15.148997 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.148984    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ef5ae74-ccbc-4e61-b92c-72d35130cb10" containerName="revoke-keys"
Oct 30 13:06:15.152700 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.152677    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"
Oct 30 13:06:15.166444 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.166415    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"]
Oct 30 13:06:15.180359 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.180327    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl\" (UID: \"b8dac73b-c32d-4fe4-971a-07fb0d8e0187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"
Oct 30 13:06:15.180567 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.180370    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75kjj\" (UniqueName: \"kubernetes.io/projected/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-kube-api-access-75kjj\") pod \"maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl\" (UID: \"b8dac73b-c32d-4fe4-971a-07fb0d8e0187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"
Oct 30 13:06:15.180757 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.180728    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl\" (UID: \"b8dac73b-c32d-4fe4-971a-07fb0d8e0187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"
Oct 30 13:06:15.281735 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.281698    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl\" (UID: \"b8dac73b-c32d-4fe4-971a-07fb0d8e0187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"
Oct 30 13:06:15.281906 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.281816    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl\" (UID: \"b8dac73b-c32d-4fe4-971a-07fb0d8e0187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"
Oct 30 13:06:15.281906 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.281841    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75kjj\" (UniqueName: \"kubernetes.io/projected/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-kube-api-access-75kjj\") pod \"maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl\" (UID: \"b8dac73b-c32d-4fe4-971a-07fb0d8e0187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"
Oct 30 13:06:15.282723 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.282693    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl\" (UID: \"b8dac73b-c32d-4fe4-971a-07fb0d8e0187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"
Oct 30 13:06:15.284327 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.284300    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl\" (UID: \"b8dac73b-c32d-4fe4-971a-07fb0d8e0187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"
Oct 30 13:06:15.302436 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.302385    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75kjj\" (UniqueName: \"kubernetes.io/projected/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-kube-api-access-75kjj\") pod \"maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl\" (UID: \"b8dac73b-c32d-4fe4-971a-07fb0d8e0187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"
Oct 30 13:06:15.329109 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.329084    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"
Oct 30 13:06:15.329229 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.329104    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j" event={"ID":"0ef5ae74-ccbc-4e61-b92c-72d35130cb10","Type":"ContainerDied","Data":"07af8ef5bdbd68b18a81c7393bbea5f2b23635e9044f063983d3d0dd66813eca"}
Oct 30 13:06:15.329229 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.329165    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07af8ef5bdbd68b18a81c7393bbea5f2b23635e9044f063983d3d0dd66813eca"
Oct 30 13:06:15.468140 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.468063    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"
Oct 30 13:06:15.713822 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.713791    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:15.714248 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.714108    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:15.714946 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.714917    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:06:15.775967 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:06:15.775023    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dac73b_c32d_4fe4_971a_07fb0d8e0187.slice/crio-f6800007bdf62f87c6de2ece632900bf63d7c4e9517aeb51b82dea39ca6de505 WatchSource:0}: Error finding container f6800007bdf62f87c6de2ece632900bf63d7c4e9517aeb51b82dea39ca6de505: Status 404 returned error can't find the container with id f6800007bdf62f87c6de2ece632900bf63d7c4e9517aeb51b82dea39ca6de505
Oct 30 13:06:15.776183 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:15.776046    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"]
Oct 30 13:06:15.995817 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:15.995778    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice/crio-eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:06:16.338268 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:16.338233    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl" event={"ID":"b8dac73b-c32d-4fe4-971a-07fb0d8e0187","Type":"ContainerStarted","Data":"b5badc9b86b354ac0e229cd15bae4d6b9f3b34b668976b6e385b01260e8fde43"}
Oct 30 13:06:16.338442 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:16.338297    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl" event={"ID":"b8dac73b-c32d-4fe4-971a-07fb0d8e0187","Type":"ContainerStarted","Data":"f6800007bdf62f87c6de2ece632900bf63d7c4e9517aeb51b82dea39ca6de505"}
Oct 30 13:06:16.380834 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:16.380740    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl" podStartSLOduration=1.380719902 podStartE2EDuration="1.380719902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:16.37567117 +0000 UTC m=+1764.899955400" watchObservedRunningTime="2026-10-30 13:06:16.380719902 +0000 UTC m=+1764.905004123"
Oct 30 13:06:16.427614 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:16.427574    5837 patch_prober.go:29] interesting pod/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:06:16.427755 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:16.427624    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:06:17.347330 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:17.347298    5837 generic.go:356] "Generic (PLEG): container finished" podID="b8dac73b-c32d-4fe4-971a-07fb0d8e0187" containerID="b5badc9b86b354ac0e229cd15bae4d6b9f3b34b668976b6e385b01260e8fde43" exitCode=0
Oct 30 13:06:17.347767 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:17.347355    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl" event={"ID":"b8dac73b-c32d-4fe4-971a-07fb0d8e0187","Type":"ContainerDied","Data":"b5badc9b86b354ac0e229cd15bae4d6b9f3b34b668976b6e385b01260e8fde43"}
Oct 30 13:06:17.573476 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:17.573435    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice/crio-eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104\": RecentStats: unable to find data in memory cache]"
Oct 30 13:06:18.552886 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:18.552858    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"
Oct 30 13:06:18.672897 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:18.672807    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-maas-api-xks-ca\") pod \"b8dac73b-c32d-4fe4-971a-07fb0d8e0187\" (UID: \"b8dac73b-c32d-4fe4-971a-07fb0d8e0187\") "
Oct 30 13:06:18.672897 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:18.672896    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-maas-api-openshift-ca\") pod \"b8dac73b-c32d-4fe4-971a-07fb0d8e0187\" (UID: \"b8dac73b-c32d-4fe4-971a-07fb0d8e0187\") "
Oct 30 13:06:18.673196 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:18.673096    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-kube-api-access-75kjj\" (UniqueName: \"kubernetes.io/projected/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-kube-api-access-75kjj\") pod \"b8dac73b-c32d-4fe4-971a-07fb0d8e0187\" (UID: \"b8dac73b-c32d-4fe4-971a-07fb0d8e0187\") "
Oct 30 13:06:18.673255 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:18.673233    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-maas-api-openshift-ca" pod "b8dac73b-c32d-4fe4-971a-07fb0d8e0187" (UID: "b8dac73b-c32d-4fe4-971a-07fb0d8e0187"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:18.673741 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:18.673658    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:18.675069 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:18.675040    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-maas-api-xks-ca" pod "b8dac73b-c32d-4fe4-971a-07fb0d8e0187" (UID: "b8dac73b-c32d-4fe4-971a-07fb0d8e0187"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:18.675268 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:18.675244    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-kube-api-access-75kjj" pod "b8dac73b-c32d-4fe4-971a-07fb0d8e0187" (UID: "b8dac73b-c32d-4fe4-971a-07fb0d8e0187"). InnerVolumeSpecName "kube-api-access-75kjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:18.775067 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:18.775037    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75kjj\" (UniqueName: \"kubernetes.io/projected/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-kube-api-access-75kjj\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:18.775067 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:18.775069    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8dac73b-c32d-4fe4-971a-07fb0d8e0187-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:19.363122 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:19.363091    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"
Oct 30 13:06:19.363302 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:19.363121    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl" event={"ID":"b8dac73b-c32d-4fe4-971a-07fb0d8e0187","Type":"ContainerDied","Data":"f6800007bdf62f87c6de2ece632900bf63d7c4e9517aeb51b82dea39ca6de505"}
Oct 30 13:06:19.363302 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:19.363169    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6800007bdf62f87c6de2ece632900bf63d7c4e9517aeb51b82dea39ca6de505"
Oct 30 13:06:19.934315 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:19.934289    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:06:20.096811 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.096769    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-workload-socket\") pod \"612f5032-bf59-40f9-b043-e20395d385b2\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") "
Oct 30 13:06:20.097006 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.096850    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/612f5032-bf59-40f9-b043-e20395d385b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612f5032-bf59-40f9-b043-e20395d385b2-istio-podinfo\") pod \"612f5032-bf59-40f9-b043-e20395d385b2\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") "
Oct 30 13:06:20.097006 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.096886    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-istio-data\") pod \"612f5032-bf59-40f9-b043-e20395d385b2\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") "
Oct 30 13:06:20.097088 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.097039    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-credential-socket\") pod \"612f5032-bf59-40f9-b043-e20395d385b2\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") "
Oct 30 13:06:20.097088 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.097047    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-workload-socket" pod "612f5032-bf59-40f9-b043-e20395d385b2" (UID: "612f5032-bf59-40f9-b043-e20395d385b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:20.097088 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.097059    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612f5032-bf59-40f9-b043-e20395d385b2-kube-api-access-mzdfj\" (UniqueName: \"kubernetes.io/projected/612f5032-bf59-40f9-b043-e20395d385b2-kube-api-access-mzdfj\") pod \"612f5032-bf59-40f9-b043-e20395d385b2\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") "
Oct 30 13:06:20.097534 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.097215    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-istio-envoy\") pod \"612f5032-bf59-40f9-b043-e20395d385b2\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") "
Oct 30 13:06:20.097534 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.097248    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/612f5032-bf59-40f9-b043-e20395d385b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612f5032-bf59-40f9-b043-e20395d385b2-istiod-ca-cert\") pod \"612f5032-bf59-40f9-b043-e20395d385b2\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") "
Oct 30 13:06:20.097534 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.097248    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-istio-data" pod "612f5032-bf59-40f9-b043-e20395d385b2" (UID: "612f5032-bf59-40f9-b043-e20395d385b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:20.097534 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.097278    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-workload-certs\") pod \"612f5032-bf59-40f9-b043-e20395d385b2\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") "
Oct 30 13:06:20.097534 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.097300    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-credential-socket" pod "612f5032-bf59-40f9-b043-e20395d385b2" (UID: "612f5032-bf59-40f9-b043-e20395d385b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:20.097534 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.097367    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612f5032-bf59-40f9-b043-e20395d385b2-istio-token\" (UniqueName: \"kubernetes.io/projected/612f5032-bf59-40f9-b043-e20395d385b2-istio-token\") pod \"612f5032-bf59-40f9-b043-e20395d385b2\" (UID: \"612f5032-bf59-40f9-b043-e20395d385b2\") "
Oct 30 13:06:20.097877 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.097593    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-workload-certs" pod "612f5032-bf59-40f9-b043-e20395d385b2" (UID: "612f5032-bf59-40f9-b043-e20395d385b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:20.097877 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.097600    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612f5032-bf59-40f9-b043-e20395d385b2-istiod-ca-cert" pod "612f5032-bf59-40f9-b043-e20395d385b2" (UID: "612f5032-bf59-40f9-b043-e20395d385b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:20.098305 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.098283    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:20.098341 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.098313    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612f5032-bf59-40f9-b043-e20395d385b2-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:20.098341 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.098328    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:20.098447 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.098341    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:20.098447 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.098353    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:20.099720 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.099695    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-istio-envoy" pod "612f5032-bf59-40f9-b043-e20395d385b2" (UID: "612f5032-bf59-40f9-b043-e20395d385b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:20.099810 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.099785    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612f5032-bf59-40f9-b043-e20395d385b2-istio-token" pod "612f5032-bf59-40f9-b043-e20395d385b2" (UID: "612f5032-bf59-40f9-b043-e20395d385b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:20.099916 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.099897    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/612f5032-bf59-40f9-b043-e20395d385b2-istio-podinfo" pod "612f5032-bf59-40f9-b043-e20395d385b2" (UID: "612f5032-bf59-40f9-b043-e20395d385b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:06:20.100091 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.100072    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612f5032-bf59-40f9-b043-e20395d385b2-kube-api-access-mzdfj" pod "612f5032-bf59-40f9-b043-e20395d385b2" (UID: "612f5032-bf59-40f9-b043-e20395d385b2"). InnerVolumeSpecName "kube-api-access-mzdfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:20.199718 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.199629    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzdfj\" (UniqueName: \"kubernetes.io/projected/612f5032-bf59-40f9-b043-e20395d385b2-kube-api-access-mzdfj\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:20.199718 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.199658    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612f5032-bf59-40f9-b043-e20395d385b2-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:20.199718 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.199667    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612f5032-bf59-40f9-b043-e20395d385b2-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:20.199718 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.199677    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612f5032-bf59-40f9-b043-e20395d385b2-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:20.371780 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.371748    5837 generic.go:356] "Generic (PLEG): container finished" podID="612f5032-bf59-40f9-b043-e20395d385b2" containerID="4f5bed7edfad0ee5957c1eb14b931952d5ca8db0c6cf7d61dcd9a782ce510ced" exitCode=0
Oct 30 13:06:20.371935 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.371802    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"
Oct 30 13:06:20.371935 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.371832    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" event={"ID":"612f5032-bf59-40f9-b043-e20395d385b2","Type":"ContainerDied","Data":"4f5bed7edfad0ee5957c1eb14b931952d5ca8db0c6cf7d61dcd9a782ce510ced"}
Oct 30 13:06:20.371935 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.371913    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4" event={"ID":"612f5032-bf59-40f9-b043-e20395d385b2","Type":"ContainerDied","Data":"fe55e1b85f69522bc41ba99eccd2f74ca5003bb35d85949cd9fe0fd009251b2f"}
Oct 30 13:06:20.372080 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.371942    5837 scope.go:122] "RemoveContainer" containerID="4f5bed7edfad0ee5957c1eb14b931952d5ca8db0c6cf7d61dcd9a782ce510ced"
Oct 30 13:06:20.384356 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.384340    5837 scope.go:122] "RemoveContainer" containerID="1f685f1dee63842b48ade8a9418369c0122cade121a5ab5604ef91e408638800"
Oct 30 13:06:20.395113 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.395094    5837 scope.go:122] "RemoveContainer" containerID="4f5bed7edfad0ee5957c1eb14b931952d5ca8db0c6cf7d61dcd9a782ce510ced"
Oct 30 13:06:20.395380 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:20.395361    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f5bed7edfad0ee5957c1eb14b931952d5ca8db0c6cf7d61dcd9a782ce510ced\": container with ID starting with 4f5bed7edfad0ee5957c1eb14b931952d5ca8db0c6cf7d61dcd9a782ce510ced not found: ID does not exist" containerID="4f5bed7edfad0ee5957c1eb14b931952d5ca8db0c6cf7d61dcd9a782ce510ced"
Oct 30 13:06:20.395443 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.395389    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f5bed7edfad0ee5957c1eb14b931952d5ca8db0c6cf7d61dcd9a782ce510ced"} err="failed to get container status \"4f5bed7edfad0ee5957c1eb14b931952d5ca8db0c6cf7d61dcd9a782ce510ced\": rpc error: code = NotFound desc = could not find container \"4f5bed7edfad0ee5957c1eb14b931952d5ca8db0c6cf7d61dcd9a782ce510ced\": container with ID starting with 4f5bed7edfad0ee5957c1eb14b931952d5ca8db0c6cf7d61dcd9a782ce510ced not found: ID does not exist"
Oct 30 13:06:20.395443 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.395420    5837 scope.go:122] "RemoveContainer" containerID="1f685f1dee63842b48ade8a9418369c0122cade121a5ab5604ef91e408638800"
Oct 30 13:06:20.395657 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:20.395638    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f685f1dee63842b48ade8a9418369c0122cade121a5ab5604ef91e408638800\": container with ID starting with 1f685f1dee63842b48ade8a9418369c0122cade121a5ab5604ef91e408638800 not found: ID does not exist" containerID="1f685f1dee63842b48ade8a9418369c0122cade121a5ab5604ef91e408638800"
Oct 30 13:06:20.395723 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.395662    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f685f1dee63842b48ade8a9418369c0122cade121a5ab5604ef91e408638800"} err="failed to get container status \"1f685f1dee63842b48ade8a9418369c0122cade121a5ab5604ef91e408638800\": rpc error: code = NotFound desc = could not find container \"1f685f1dee63842b48ade8a9418369c0122cade121a5ab5604ef91e408638800\": container with ID starting with 1f685f1dee63842b48ade8a9418369c0122cade121a5ab5604ef91e408638800 not found: ID does not exist"
Oct 30 13:06:20.425635 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.425615    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"]
Oct 30 13:06:20.437496 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:20.437470    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-jq5v4"]
Oct 30 13:06:21.361723 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:21.361688    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"]
Oct 30 13:06:21.392995 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:21.392965    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"]
Oct 30 13:06:21.629476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:21.628526    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="612f5032-bf59-40f9-b043-e20395d385b2" path="/var/lib/kubelet/pods/612f5032-bf59-40f9-b043-e20395d385b2/volumes"
Oct 30 13:06:25.548634 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:25.548603    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"]
Oct 30 13:06:25.577737 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:25.577711    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"]
Oct 30 13:06:26.362622 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:26.362575    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm" podUID="61c702c3-5d1c-44d9-b02e-1fa7e139ca9c" containerName="payload-processing" containerID="cri-o://a49be8fc2aeb78b901f2ac9f4947b7612f09e20680cb493db39ae4a38145ea7a" gracePeriod=25
Oct 30 13:06:26.393527 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:26.393455    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9" podUID="1c4895db-c87e-45f5-9317-e2643409cff3" containerName="payload-pre-processing" containerID="cri-o://bb428992badd8568939c0307d90c4fa52f70eb08cc8b7541ba6531cbbd7c5963" gracePeriod=25
Oct 30 13:06:27.663868 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:27.663832    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice/crio-eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:06:28.872565 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:28.872525    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9" podUID="1c4895db-c87e-45f5-9317-e2643409cff3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:06:29.880962 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:29.880912    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm" podUID="61c702c3-5d1c-44d9-b02e-1fa7e139ca9c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:06:30.549318 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:30.549280    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9" podUID="663b46aa-7286-46ae-b760-d56bf2cf7d5d" containerName="payload-processing" containerID="cri-o://f548c1a9f2a74a54fc6b4f0654f24c40207cb77131970f7f85378f7819834572" gracePeriod=25
Oct 30 13:06:30.578297 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:30.578263    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6" podUID="8a9fbc5c-80d9-4674-b16e-dd48110d1fad" containerName="payload-pre-processing" containerID="cri-o://6c7aaed2b66c812e0fc44e7291dca93c6c4ab3e7a8f56eb87378e57068c9dfdd" gracePeriod=25
Oct 30 13:06:30.877779 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:30.877745    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"
Oct 30 13:06:30.912127 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:30.912105    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"
Oct 30 13:06:30.929633 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:30.929599    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice/crio-eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104\": RecentStats: unable to find data in memory cache]"
Oct 30 13:06:30.989201 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:30.989170    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a9fbc5c-80d9-4674-b16e-dd48110d1fad-kube-api-access-5mc9k\" (UniqueName: \"kubernetes.io/projected/8a9fbc5c-80d9-4674-b16e-dd48110d1fad-kube-api-access-5mc9k\") pod \"8a9fbc5c-80d9-4674-b16e-dd48110d1fad\" (UID: \"8a9fbc5c-80d9-4674-b16e-dd48110d1fad\") "
Oct 30 13:06:30.989386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:30.989316    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663b46aa-7286-46ae-b760-d56bf2cf7d5d-kube-api-access-db24l\" (UniqueName: \"kubernetes.io/projected/663b46aa-7286-46ae-b760-d56bf2cf7d5d-kube-api-access-db24l\") pod \"663b46aa-7286-46ae-b760-d56bf2cf7d5d\" (UID: \"663b46aa-7286-46ae-b760-d56bf2cf7d5d\") "
Oct 30 13:06:30.989596 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:30.989572    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a9fbc5c-80d9-4674-b16e-dd48110d1fad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9fbc5c-80d9-4674-b16e-dd48110d1fad-plugins-config-volume\") pod \"8a9fbc5c-80d9-4674-b16e-dd48110d1fad\" (UID: \"8a9fbc5c-80d9-4674-b16e-dd48110d1fad\") "
Oct 30 13:06:30.989703 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:30.989667    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663b46aa-7286-46ae-b760-d56bf2cf7d5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663b46aa-7286-46ae-b760-d56bf2cf7d5d-plugins-config-volume\") pod \"663b46aa-7286-46ae-b760-d56bf2cf7d5d\" (UID: \"663b46aa-7286-46ae-b760-d56bf2cf7d5d\") "
Oct 30 13:06:30.990006 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:30.989976    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9fbc5c-80d9-4674-b16e-dd48110d1fad-plugins-config-volume" pod "8a9fbc5c-80d9-4674-b16e-dd48110d1fad" (UID: "8a9fbc5c-80d9-4674-b16e-dd48110d1fad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:30.990502 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:30.990477    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663b46aa-7286-46ae-b760-d56bf2cf7d5d-plugins-config-volume" pod "663b46aa-7286-46ae-b760-d56bf2cf7d5d" (UID: "663b46aa-7286-46ae-b760-d56bf2cf7d5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:30.990631 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:30.990587    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9fbc5c-80d9-4674-b16e-dd48110d1fad-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:30.991676 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:30.991655    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663b46aa-7286-46ae-b760-d56bf2cf7d5d-kube-api-access-db24l" pod "663b46aa-7286-46ae-b760-d56bf2cf7d5d" (UID: "663b46aa-7286-46ae-b760-d56bf2cf7d5d"). InnerVolumeSpecName "kube-api-access-db24l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:30.992115 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:30.992097    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9fbc5c-80d9-4674-b16e-dd48110d1fad-kube-api-access-5mc9k" pod "8a9fbc5c-80d9-4674-b16e-dd48110d1fad" (UID: "8a9fbc5c-80d9-4674-b16e-dd48110d1fad"). InnerVolumeSpecName "kube-api-access-5mc9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:31.092716 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.092635    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663b46aa-7286-46ae-b760-d56bf2cf7d5d-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.092716 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.092678    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mc9k\" (UniqueName: \"kubernetes.io/projected/8a9fbc5c-80d9-4674-b16e-dd48110d1fad-kube-api-access-5mc9k\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.092716 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.092695    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db24l\" (UniqueName: \"kubernetes.io/projected/663b46aa-7286-46ae-b760-d56bf2cf7d5d-kube-api-access-db24l\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.465887 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.465801    5837 generic.go:356] "Generic (PLEG): container finished" podID="663b46aa-7286-46ae-b760-d56bf2cf7d5d" containerID="f548c1a9f2a74a54fc6b4f0654f24c40207cb77131970f7f85378f7819834572" exitCode=0
Oct 30 13:06:31.465887 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.465842    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9" event={"ID":"663b46aa-7286-46ae-b760-d56bf2cf7d5d","Type":"ContainerDied","Data":"f548c1a9f2a74a54fc6b4f0654f24c40207cb77131970f7f85378f7819834572"}
Oct 30 13:06:31.466192 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.465859    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"
Oct 30 13:06:31.466192 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.465949    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9" event={"ID":"663b46aa-7286-46ae-b760-d56bf2cf7d5d","Type":"ContainerDied","Data":"0dd7388c924e473ba45fdfac3a1e84655ddec88d36c8683a9217a536bc6d8c1a"}
Oct 30 13:06:31.466192 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.465980    5837 scope.go:122] "RemoveContainer" containerID="f548c1a9f2a74a54fc6b4f0654f24c40207cb77131970f7f85378f7819834572"
Oct 30 13:06:31.467594 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.467570    5837 generic.go:356] "Generic (PLEG): container finished" podID="8a9fbc5c-80d9-4674-b16e-dd48110d1fad" containerID="6c7aaed2b66c812e0fc44e7291dca93c6c4ab3e7a8f56eb87378e57068c9dfdd" exitCode=0
Oct 30 13:06:31.467723 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.467609    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"
Oct 30 13:06:31.467723 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.467626    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6" event={"ID":"8a9fbc5c-80d9-4674-b16e-dd48110d1fad","Type":"ContainerDied","Data":"6c7aaed2b66c812e0fc44e7291dca93c6c4ab3e7a8f56eb87378e57068c9dfdd"}
Oct 30 13:06:31.467723 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.467667    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6" event={"ID":"8a9fbc5c-80d9-4674-b16e-dd48110d1fad","Type":"ContainerDied","Data":"b688cf1fc9b20b4763d01b4b9bb22b3d130638cec4bbc04d255d37bba43a8f74"}
Oct 30 13:06:31.477516 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.477500    5837 scope.go:122] "RemoveContainer" containerID="f548c1a9f2a74a54fc6b4f0654f24c40207cb77131970f7f85378f7819834572"
Oct 30 13:06:31.477818 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:31.477796    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f548c1a9f2a74a54fc6b4f0654f24c40207cb77131970f7f85378f7819834572\": container with ID starting with f548c1a9f2a74a54fc6b4f0654f24c40207cb77131970f7f85378f7819834572 not found: ID does not exist" containerID="f548c1a9f2a74a54fc6b4f0654f24c40207cb77131970f7f85378f7819834572"
Oct 30 13:06:31.477891 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.477826    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f548c1a9f2a74a54fc6b4f0654f24c40207cb77131970f7f85378f7819834572"} err="failed to get container status \"f548c1a9f2a74a54fc6b4f0654f24c40207cb77131970f7f85378f7819834572\": rpc error: code = NotFound desc = could not find container \"f548c1a9f2a74a54fc6b4f0654f24c40207cb77131970f7f85378f7819834572\": container with ID starting with f548c1a9f2a74a54fc6b4f0654f24c40207cb77131970f7f85378f7819834572 not found: ID does not exist"
Oct 30 13:06:31.477891 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.477842    5837 scope.go:122] "RemoveContainer" containerID="6c7aaed2b66c812e0fc44e7291dca93c6c4ab3e7a8f56eb87378e57068c9dfdd"
Oct 30 13:06:31.487563 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.487548    5837 scope.go:122] "RemoveContainer" containerID="6c7aaed2b66c812e0fc44e7291dca93c6c4ab3e7a8f56eb87378e57068c9dfdd"
Oct 30 13:06:31.487827 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:31.487808    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7aaed2b66c812e0fc44e7291dca93c6c4ab3e7a8f56eb87378e57068c9dfdd\": container with ID starting with 6c7aaed2b66c812e0fc44e7291dca93c6c4ab3e7a8f56eb87378e57068c9dfdd not found: ID does not exist" containerID="6c7aaed2b66c812e0fc44e7291dca93c6c4ab3e7a8f56eb87378e57068c9dfdd"
Oct 30 13:06:31.487911 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.487837    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7aaed2b66c812e0fc44e7291dca93c6c4ab3e7a8f56eb87378e57068c9dfdd"} err="failed to get container status \"6c7aaed2b66c812e0fc44e7291dca93c6c4ab3e7a8f56eb87378e57068c9dfdd\": rpc error: code = NotFound desc = could not find container \"6c7aaed2b66c812e0fc44e7291dca93c6c4ab3e7a8f56eb87378e57068c9dfdd\": container with ID starting with 6c7aaed2b66c812e0fc44e7291dca93c6c4ab3e7a8f56eb87378e57068c9dfdd not found: ID does not exist"
Oct 30 13:06:31.521124 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.521101    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"]
Oct 30 13:06:31.525114 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.525092    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-666577-f6d486d8c-zpht6"]
Oct 30 13:06:31.569488 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.569467    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"]
Oct 30 13:06:31.574249 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.574227    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-666577-764dc95b64-l8hx9"]
Oct 30 13:06:31.605878 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.605851    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663b46aa-7286-46ae-b760-d56bf2cf7d5d" path="/var/lib/kubelet/pods/663b46aa-7286-46ae-b760-d56bf2cf7d5d/volumes"
Oct 30 13:06:31.606254 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:31.606239    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a9fbc5c-80d9-4674-b16e-dd48110d1fad" path="/var/lib/kubelet/pods/8a9fbc5c-80d9-4674-b16e-dd48110d1fad/volumes"
Oct 30 13:06:33.857920 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:33.857880    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"]
Oct 30 13:06:33.862534 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:33.862510    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-666577-9kwsl"]
Oct 30 13:06:35.427882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.427853    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"]
Oct 30 13:06:35.428510 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.428496    5837 cpu_manager.go:414] "containerMap: removing container" podUID="663b46aa-7286-46ae-b760-d56bf2cf7d5d" containerName="payload-processing"
Oct 30 13:06:35.428561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.428510    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663b46aa-7286-46ae-b760-d56bf2cf7d5d" containerName="payload-processing"
Oct 30 13:06:35.428561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.428521    5837 cpu_manager.go:414] "containerMap: removing container" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy"
Oct 30 13:06:35.428561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.428526    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy"
Oct 30 13:06:35.428561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.428534    5837 cpu_manager.go:414] "containerMap: removing container" podUID="8a9fbc5c-80d9-4674-b16e-dd48110d1fad" containerName="payload-pre-processing"
Oct 30 13:06:35.428561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.428539    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a9fbc5c-80d9-4674-b16e-dd48110d1fad" containerName="payload-pre-processing"
Oct 30 13:06:35.428561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.428556    5837 cpu_manager.go:414] "containerMap: removing container" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy"
Oct 30 13:06:35.428561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.428560    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy"
Oct 30 13:06:35.428784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.428570    5837 cpu_manager.go:414] "containerMap: removing container" podUID="b8dac73b-c32d-4fe4-971a-07fb0d8e0187" containerName="revoke-keys"
Oct 30 13:06:35.428784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.428575    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8dac73b-c32d-4fe4-971a-07fb0d8e0187" containerName="revoke-keys"
Oct 30 13:06:35.428784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.428664    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy"
Oct 30 13:06:35.428784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.428672    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8dac73b-c32d-4fe4-971a-07fb0d8e0187" containerName="revoke-keys"
Oct 30 13:06:35.428784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.428688    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="663b46aa-7286-46ae-b760-d56bf2cf7d5d" containerName="payload-processing"
Oct 30 13:06:35.428784 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.428696    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a9fbc5c-80d9-4674-b16e-dd48110d1fad" containerName="payload-pre-processing"
Oct 30 13:06:35.432477 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.432461    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"
Oct 30 13:06:35.446683 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.444671    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"]
Oct 30 13:06:35.446683 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.446639    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="612f5032-bf59-40f9-b043-e20395d385b2" containerName="istio-proxy"
Oct 30 13:06:35.452661 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.452627    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:06:35.452871 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.452717    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:06:35.452871 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.452769    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:06:35.459695 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.459654    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:06:35.460235 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.460208    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"]
Oct 30 13:06:35.460354 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.460339    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"
Oct 30 13:06:35.469140 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.469110    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"]
Oct 30 13:06:35.586457 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.586371    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf688ee-b349-40af-a2cc-14877d335e34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54e7adec-jwpqb\" (UID: \"1cf688ee-b349-40af-a2cc-14877d335e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"
Oct 30 13:06:35.586674 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.586535    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6879e6-651c-4a62-b0fa-511232557e12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e32c90f-hzj97\" (UID: \"5c6879e6-651c-4a62-b0fa-511232557e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"
Oct 30 13:06:35.586674 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.586579    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf688ee-b349-40af-a2cc-14877d335e34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54e7adec-jwpqb\" (UID: \"1cf688ee-b349-40af-a2cc-14877d335e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"
Oct 30 13:06:35.586796 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.586711    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmhm\" (UniqueName: \"kubernetes.io/projected/1cf688ee-b349-40af-a2cc-14877d335e34-kube-api-access-8nmhm\") pod \"maas-api-revoke-sub-54e7adec-jwpqb\" (UID: \"1cf688ee-b349-40af-a2cc-14877d335e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"
Oct 30 13:06:35.586888 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.586870    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6879e6-651c-4a62-b0fa-511232557e12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e32c90f-hzj97\" (UID: \"5c6879e6-651c-4a62-b0fa-511232557e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"
Oct 30 13:06:35.587040 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.587022    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzh8k\" (UniqueName: \"kubernetes.io/projected/5c6879e6-651c-4a62-b0fa-511232557e12-kube-api-access-tzh8k\") pod \"maas-api-revoke-sub-9e32c90f-hzj97\" (UID: \"5c6879e6-651c-4a62-b0fa-511232557e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"
Oct 30 13:06:35.606370 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.606340    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8dac73b-c32d-4fe4-971a-07fb0d8e0187" path="/var/lib/kubelet/pods/b8dac73b-c32d-4fe4-971a-07fb0d8e0187/volumes"
Oct 30 13:06:35.688488 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.688363    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf688ee-b349-40af-a2cc-14877d335e34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54e7adec-jwpqb\" (UID: \"1cf688ee-b349-40af-a2cc-14877d335e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"
Oct 30 13:06:35.688657 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.688492    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6879e6-651c-4a62-b0fa-511232557e12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e32c90f-hzj97\" (UID: \"5c6879e6-651c-4a62-b0fa-511232557e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"
Oct 30 13:06:35.688657 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.688530    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf688ee-b349-40af-a2cc-14877d335e34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54e7adec-jwpqb\" (UID: \"1cf688ee-b349-40af-a2cc-14877d335e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"
Oct 30 13:06:35.688657 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.688580    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmhm\" (UniqueName: \"kubernetes.io/projected/1cf688ee-b349-40af-a2cc-14877d335e34-kube-api-access-8nmhm\") pod \"maas-api-revoke-sub-54e7adec-jwpqb\" (UID: \"1cf688ee-b349-40af-a2cc-14877d335e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"
Oct 30 13:06:35.688759 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.688694    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6879e6-651c-4a62-b0fa-511232557e12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e32c90f-hzj97\" (UID: \"5c6879e6-651c-4a62-b0fa-511232557e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"
Oct 30 13:06:35.688849 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.688834    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzh8k\" (UniqueName: \"kubernetes.io/projected/5c6879e6-651c-4a62-b0fa-511232557e12-kube-api-access-tzh8k\") pod \"maas-api-revoke-sub-9e32c90f-hzj97\" (UID: \"5c6879e6-651c-4a62-b0fa-511232557e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"
Oct 30 13:06:35.689176 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.689146    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf688ee-b349-40af-a2cc-14877d335e34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54e7adec-jwpqb\" (UID: \"1cf688ee-b349-40af-a2cc-14877d335e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"
Oct 30 13:06:35.689492 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.689468    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6879e6-651c-4a62-b0fa-511232557e12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e32c90f-hzj97\" (UID: \"5c6879e6-651c-4a62-b0fa-511232557e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"
Oct 30 13:06:35.691077 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.691052    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf688ee-b349-40af-a2cc-14877d335e34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54e7adec-jwpqb\" (UID: \"1cf688ee-b349-40af-a2cc-14877d335e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"
Oct 30 13:06:35.691077 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.691063    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6879e6-651c-4a62-b0fa-511232557e12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e32c90f-hzj97\" (UID: \"5c6879e6-651c-4a62-b0fa-511232557e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"
Oct 30 13:06:35.710862 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.710832    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmhm\" (UniqueName: \"kubernetes.io/projected/1cf688ee-b349-40af-a2cc-14877d335e34-kube-api-access-8nmhm\") pod \"maas-api-revoke-sub-54e7adec-jwpqb\" (UID: \"1cf688ee-b349-40af-a2cc-14877d335e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"
Oct 30 13:06:35.711033 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.710917    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzh8k\" (UniqueName: \"kubernetes.io/projected/5c6879e6-651c-4a62-b0fa-511232557e12-kube-api-access-tzh8k\") pod \"maas-api-revoke-sub-9e32c90f-hzj97\" (UID: \"5c6879e6-651c-4a62-b0fa-511232557e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"
Oct 30 13:06:35.761297 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.761274    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"
Oct 30 13:06:35.786300 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:35.786279    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"
Oct 30 13:06:36.067833 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:06:36.067803    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6879e6_651c_4a62_b0fa_511232557e12.slice/crio-15f5a8a4ad5837db168f47c4f1bba602a4157622b4e5dd4c9c835b504cacc3fa WatchSource:0}: Error finding container 15f5a8a4ad5837db168f47c4f1bba602a4157622b4e5dd4c9c835b504cacc3fa: Status 404 returned error can't find the container with id 15f5a8a4ad5837db168f47c4f1bba602a4157622b4e5dd4c9c835b504cacc3fa
Oct 30 13:06:36.068125 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:36.068097    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"]
Oct 30 13:06:36.102117 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:36.102089    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"]
Oct 30 13:06:36.102550 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:06:36.102523    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf688ee_b349_40af_a2cc_14877d335e34.slice/crio-7671e4ba3f8cfa5ca56efaec2e50c631887a5f181caac8e0545032935d7e0cc4 WatchSource:0}: Error finding container 7671e4ba3f8cfa5ca56efaec2e50c631887a5f181caac8e0545032935d7e0cc4: Status 404 returned error can't find the container with id 7671e4ba3f8cfa5ca56efaec2e50c631887a5f181caac8e0545032935d7e0cc4
Oct 30 13:06:36.517351 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:36.517314    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb" event={"ID":"1cf688ee-b349-40af-a2cc-14877d335e34","Type":"ContainerStarted","Data":"090622ae7e09c017a7e8f76b431b0e1601e0fd79f1a1dc0ec549cba62dbc13cf"}
Oct 30 13:06:36.517806 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:36.517377    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb" event={"ID":"1cf688ee-b349-40af-a2cc-14877d335e34","Type":"ContainerStarted","Data":"7671e4ba3f8cfa5ca56efaec2e50c631887a5f181caac8e0545032935d7e0cc4"}
Oct 30 13:06:36.519181 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:36.519158    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97" event={"ID":"5c6879e6-651c-4a62-b0fa-511232557e12","Type":"ContainerStarted","Data":"0c312a94b34d83473704a30780b4722d2561765f08d39413f81c5c1ac020a568"}
Oct 30 13:06:36.519305 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:36.519202    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97" event={"ID":"5c6879e6-651c-4a62-b0fa-511232557e12","Type":"ContainerStarted","Data":"15f5a8a4ad5837db168f47c4f1bba602a4157622b4e5dd4c9c835b504cacc3fa"}
Oct 30 13:06:36.556004 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:36.555959    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb" podStartSLOduration=1.555945966 podStartE2EDuration="1.555945966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:36.555029711 +0000 UTC m=+1785.079313929" watchObservedRunningTime="2026-10-30 13:06:36.555945966 +0000 UTC m=+1785.080230182"
Oct 30 13:06:36.594374 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:36.594321    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97" podStartSLOduration=1.594309889 podStartE2EDuration="1.594309889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:36.59221593 +0000 UTC m=+1785.116500151" watchObservedRunningTime="2026-10-30 13:06:36.594309889 +0000 UTC m=+1785.118594110"
Oct 30 13:06:37.528079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:37.528045    5837 generic.go:356] "Generic (PLEG): container finished" podID="1cf688ee-b349-40af-a2cc-14877d335e34" containerID="090622ae7e09c017a7e8f76b431b0e1601e0fd79f1a1dc0ec549cba62dbc13cf" exitCode=0
Oct 30 13:06:37.528532 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:37.528085    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb" event={"ID":"1cf688ee-b349-40af-a2cc-14877d335e34","Type":"ContainerDied","Data":"090622ae7e09c017a7e8f76b431b0e1601e0fd79f1a1dc0ec549cba62dbc13cf"}
Oct 30 13:06:37.529980 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:37.529958    5837 generic.go:356] "Generic (PLEG): container finished" podID="5c6879e6-651c-4a62-b0fa-511232557e12" containerID="0c312a94b34d83473704a30780b4722d2561765f08d39413f81c5c1ac020a568" exitCode=0
Oct 30 13:06:37.530127 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:37.530000    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97" event={"ID":"5c6879e6-651c-4a62-b0fa-511232557e12","Type":"ContainerDied","Data":"0c312a94b34d83473704a30780b4722d2561765f08d39413f81c5c1ac020a568"}
Oct 30 13:06:37.739343 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:37.739297    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice/crio-eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104\": RecentStats: unable to find data in memory cache]"
Oct 30 13:06:38.775474 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.775448    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"
Oct 30 13:06:38.779662 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.779640    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"
Oct 30 13:06:38.868801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.868704    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cf688ee-b349-40af-a2cc-14877d335e34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf688ee-b349-40af-a2cc-14877d335e34-maas-api-xks-ca\") pod \"1cf688ee-b349-40af-a2cc-14877d335e34\" (UID: \"1cf688ee-b349-40af-a2cc-14877d335e34\") "
Oct 30 13:06:38.868801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.868776    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf688ee-b349-40af-a2cc-14877d335e34-kube-api-access-8nmhm\" (UniqueName: \"kubernetes.io/projected/1cf688ee-b349-40af-a2cc-14877d335e34-kube-api-access-8nmhm\") pod \"1cf688ee-b349-40af-a2cc-14877d335e34\" (UID: \"1cf688ee-b349-40af-a2cc-14877d335e34\") "
Oct 30 13:06:38.869044 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.868984    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c6879e6-651c-4a62-b0fa-511232557e12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6879e6-651c-4a62-b0fa-511232557e12-maas-api-xks-ca\") pod \"5c6879e6-651c-4a62-b0fa-511232557e12\" (UID: \"5c6879e6-651c-4a62-b0fa-511232557e12\") "
Oct 30 13:06:38.869109 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.869055    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf688ee-b349-40af-a2cc-14877d335e34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf688ee-b349-40af-a2cc-14877d335e34-maas-api-openshift-ca\") pod \"1cf688ee-b349-40af-a2cc-14877d335e34\" (UID: \"1cf688ee-b349-40af-a2cc-14877d335e34\") "
Oct 30 13:06:38.869194 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.869177    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6879e6-651c-4a62-b0fa-511232557e12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6879e6-651c-4a62-b0fa-511232557e12-maas-api-openshift-ca\") pod \"5c6879e6-651c-4a62-b0fa-511232557e12\" (UID: \"5c6879e6-651c-4a62-b0fa-511232557e12\") "
Oct 30 13:06:38.869249 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.869214    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6879e6-651c-4a62-b0fa-511232557e12-kube-api-access-tzh8k\" (UniqueName: \"kubernetes.io/projected/5c6879e6-651c-4a62-b0fa-511232557e12-kube-api-access-tzh8k\") pod \"5c6879e6-651c-4a62-b0fa-511232557e12\" (UID: \"5c6879e6-651c-4a62-b0fa-511232557e12\") "
Oct 30 13:06:38.870015 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.869487    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf688ee-b349-40af-a2cc-14877d335e34-maas-api-openshift-ca" pod "1cf688ee-b349-40af-a2cc-14877d335e34" (UID: "1cf688ee-b349-40af-a2cc-14877d335e34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:38.870138 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.869856    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6879e6-651c-4a62-b0fa-511232557e12-maas-api-openshift-ca" pod "5c6879e6-651c-4a62-b0fa-511232557e12" (UID: "5c6879e6-651c-4a62-b0fa-511232557e12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:38.872598 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.872569    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf688ee-b349-40af-a2cc-14877d335e34-maas-api-xks-ca" pod "1cf688ee-b349-40af-a2cc-14877d335e34" (UID: "1cf688ee-b349-40af-a2cc-14877d335e34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:38.872720 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.872684    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c6879e6-651c-4a62-b0fa-511232557e12-maas-api-xks-ca" pod "5c6879e6-651c-4a62-b0fa-511232557e12" (UID: "5c6879e6-651c-4a62-b0fa-511232557e12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:38.872969 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.872936    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf688ee-b349-40af-a2cc-14877d335e34-kube-api-access-8nmhm" pod "1cf688ee-b349-40af-a2cc-14877d335e34" (UID: "1cf688ee-b349-40af-a2cc-14877d335e34"). InnerVolumeSpecName "kube-api-access-8nmhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:38.873063 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.873027    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6879e6-651c-4a62-b0fa-511232557e12-kube-api-access-tzh8k" pod "5c6879e6-651c-4a62-b0fa-511232557e12" (UID: "5c6879e6-651c-4a62-b0fa-511232557e12"). InnerVolumeSpecName "kube-api-access-tzh8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:38.873434 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.873364    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9" podUID="1c4895db-c87e-45f5-9317-e2643409cff3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:06:38.971875 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.971824    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6879e6-651c-4a62-b0fa-511232557e12-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:38.971875 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.971869    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf688ee-b349-40af-a2cc-14877d335e34-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:38.971875 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.971886    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6879e6-651c-4a62-b0fa-511232557e12-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:38.972140 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.971900    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzh8k\" (UniqueName: \"kubernetes.io/projected/5c6879e6-651c-4a62-b0fa-511232557e12-kube-api-access-tzh8k\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:38.972140 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.971913    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf688ee-b349-40af-a2cc-14877d335e34-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:38.972140 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:38.971926    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nmhm\" (UniqueName: \"kubernetes.io/projected/1cf688ee-b349-40af-a2cc-14877d335e34-kube-api-access-8nmhm\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:39.555012 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:39.554978    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"
Oct 30 13:06:39.555212 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:39.555009    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97" event={"ID":"5c6879e6-651c-4a62-b0fa-511232557e12","Type":"ContainerDied","Data":"15f5a8a4ad5837db168f47c4f1bba602a4157622b4e5dd4c9c835b504cacc3fa"}
Oct 30 13:06:39.555212 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:39.555094    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15f5a8a4ad5837db168f47c4f1bba602a4157622b4e5dd4c9c835b504cacc3fa"
Oct 30 13:06:39.556624 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:39.556605    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"
Oct 30 13:06:39.556752 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:39.556642    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb" event={"ID":"1cf688ee-b349-40af-a2cc-14877d335e34","Type":"ContainerDied","Data":"7671e4ba3f8cfa5ca56efaec2e50c631887a5f181caac8e0545032935d7e0cc4"}
Oct 30 13:06:39.556752 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:39.556688    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7671e4ba3f8cfa5ca56efaec2e50c631887a5f181caac8e0545032935d7e0cc4"
Oct 30 13:06:39.881634 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:39.881544    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm" podUID="61c702c3-5d1c-44d9-b02e-1fa7e139ca9c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:06:40.883195 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:40.883165    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"]
Oct 30 13:06:40.883662 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:40.883362    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs" podUID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerName="istio-proxy" containerID="cri-o://1e137026135c14ff1fa6c5a001f792310a8472789567563ac88b32a81ebb1362" gracePeriod=30
Oct 30 13:06:41.823948 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.823918    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"]
Oct 30 13:06:41.824682 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.824667    5837 cpu_manager.go:414] "containerMap: removing container" podUID="1cf688ee-b349-40af-a2cc-14877d335e34" containerName="revoke-keys"
Oct 30 13:06:41.824737 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.824684    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf688ee-b349-40af-a2cc-14877d335e34" containerName="revoke-keys"
Oct 30 13:06:41.824737 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.824697    5837 cpu_manager.go:414] "containerMap: removing container" podUID="5c6879e6-651c-4a62-b0fa-511232557e12" containerName="revoke-keys"
Oct 30 13:06:41.824737 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.824701    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6879e6-651c-4a62-b0fa-511232557e12" containerName="revoke-keys"
Oct 30 13:06:41.824837 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.824817    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf688ee-b349-40af-a2cc-14877d335e34" containerName="revoke-keys"
Oct 30 13:06:41.824837 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.824827    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6879e6-651c-4a62-b0fa-511232557e12" containerName="revoke-keys"
Oct 30 13:06:41.828520 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.828502    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"
Oct 30 13:06:41.844528 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.844497    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"]
Oct 30 13:06:41.851692 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.851630    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:06:41.851692 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.851644    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:06:41.851692 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.851670    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:06:41.854125 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.854102    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:06:41.931069 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.931027    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f859f6e3-d493-48eb-9a57-49cf2199a46c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw\" (UID: \"f859f6e3-d493-48eb-9a57-49cf2199a46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"
Oct 30 13:06:41.931501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.931101    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwxmp\" (UniqueName: \"kubernetes.io/projected/f859f6e3-d493-48eb-9a57-49cf2199a46c-kube-api-access-mwxmp\") pod \"maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw\" (UID: \"f859f6e3-d493-48eb-9a57-49cf2199a46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"
Oct 30 13:06:41.931501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:41.931217    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f859f6e3-d493-48eb-9a57-49cf2199a46c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw\" (UID: \"f859f6e3-d493-48eb-9a57-49cf2199a46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"
Oct 30 13:06:42.032190 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:42.032157    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwxmp\" (UniqueName: \"kubernetes.io/projected/f859f6e3-d493-48eb-9a57-49cf2199a46c-kube-api-access-mwxmp\") pod \"maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw\" (UID: \"f859f6e3-d493-48eb-9a57-49cf2199a46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"
Oct 30 13:06:42.032385 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:42.032247    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f859f6e3-d493-48eb-9a57-49cf2199a46c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw\" (UID: \"f859f6e3-d493-48eb-9a57-49cf2199a46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"
Oct 30 13:06:42.032533 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:42.032515    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f859f6e3-d493-48eb-9a57-49cf2199a46c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw\" (UID: \"f859f6e3-d493-48eb-9a57-49cf2199a46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"
Oct 30 13:06:42.033088 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:42.033068    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f859f6e3-d493-48eb-9a57-49cf2199a46c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw\" (UID: \"f859f6e3-d493-48eb-9a57-49cf2199a46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"
Oct 30 13:06:42.034735 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:42.034715    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f859f6e3-d493-48eb-9a57-49cf2199a46c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw\" (UID: \"f859f6e3-d493-48eb-9a57-49cf2199a46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"
Oct 30 13:06:42.050935 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:42.050911    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwxmp\" (UniqueName: \"kubernetes.io/projected/f859f6e3-d493-48eb-9a57-49cf2199a46c-kube-api-access-mwxmp\") pod \"maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw\" (UID: \"f859f6e3-d493-48eb-9a57-49cf2199a46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"
Oct 30 13:06:42.142570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:42.142499    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"
Oct 30 13:06:42.505647 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:42.505616    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"]
Oct 30 13:06:42.521698 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:06:42.521664    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf859f6e3_d493_48eb_9a57_49cf2199a46c.slice/crio-0435993e49e5a4ed0453758c9c537b1f7b044d8b2989c828910f8c00bd12122c WatchSource:0}: Error finding container 0435993e49e5a4ed0453758c9c537b1f7b044d8b2989c828910f8c00bd12122c: Status 404 returned error can't find the container with id 0435993e49e5a4ed0453758c9c537b1f7b044d8b2989c828910f8c00bd12122c
Oct 30 13:06:42.551946 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:42.551922    5837 patch_prober.go:29] interesting pod/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 13:06:42.552073 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:42.551972    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs" podUID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:06:42.620221 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:42.620180    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw" event={"ID":"f859f6e3-d493-48eb-9a57-49cf2199a46c","Type":"ContainerStarted","Data":"0435993e49e5a4ed0453758c9c537b1f7b044d8b2989c828910f8c00bd12122c"}
Oct 30 13:06:43.628869 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:43.628836    5837 generic.go:356] "Generic (PLEG): container finished" podID="f859f6e3-d493-48eb-9a57-49cf2199a46c" containerID="7ff2b334e7b8aff79b6d94b7e13f3a048ccfb8e7e12d42a94d085493b27e3069" exitCode=0
Oct 30 13:06:43.629235 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:43.628921    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw" event={"ID":"f859f6e3-d493-48eb-9a57-49cf2199a46c","Type":"ContainerDied","Data":"7ff2b334e7b8aff79b6d94b7e13f3a048ccfb8e7e12d42a94d085493b27e3069"}
Oct 30 13:06:44.867274 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:44.867249    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"
Oct 30 13:06:44.974979 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:44.974898    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f859f6e3-d493-48eb-9a57-49cf2199a46c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f859f6e3-d493-48eb-9a57-49cf2199a46c-maas-api-openshift-ca\") pod \"f859f6e3-d493-48eb-9a57-49cf2199a46c\" (UID: \"f859f6e3-d493-48eb-9a57-49cf2199a46c\") "
Oct 30 13:06:44.975155 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:44.975083    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f859f6e3-d493-48eb-9a57-49cf2199a46c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f859f6e3-d493-48eb-9a57-49cf2199a46c-maas-api-xks-ca\") pod \"f859f6e3-d493-48eb-9a57-49cf2199a46c\" (UID: \"f859f6e3-d493-48eb-9a57-49cf2199a46c\") "
Oct 30 13:06:44.975225 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:44.975187    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f859f6e3-d493-48eb-9a57-49cf2199a46c-kube-api-access-mwxmp\" (UniqueName: \"kubernetes.io/projected/f859f6e3-d493-48eb-9a57-49cf2199a46c-kube-api-access-mwxmp\") pod \"f859f6e3-d493-48eb-9a57-49cf2199a46c\" (UID: \"f859f6e3-d493-48eb-9a57-49cf2199a46c\") "
Oct 30 13:06:44.975327 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:44.975304    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f859f6e3-d493-48eb-9a57-49cf2199a46c-maas-api-openshift-ca" pod "f859f6e3-d493-48eb-9a57-49cf2199a46c" (UID: "f859f6e3-d493-48eb-9a57-49cf2199a46c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:44.975705 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:44.975685    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f859f6e3-d493-48eb-9a57-49cf2199a46c-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:44.977449 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:44.977427    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f859f6e3-d493-48eb-9a57-49cf2199a46c-maas-api-xks-ca" pod "f859f6e3-d493-48eb-9a57-49cf2199a46c" (UID: "f859f6e3-d493-48eb-9a57-49cf2199a46c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:44.977525 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:44.977511    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f859f6e3-d493-48eb-9a57-49cf2199a46c-kube-api-access-mwxmp" pod "f859f6e3-d493-48eb-9a57-49cf2199a46c" (UID: "f859f6e3-d493-48eb-9a57-49cf2199a46c"). InnerVolumeSpecName "kube-api-access-mwxmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:45.076914 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:45.076891    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f859f6e3-d493-48eb-9a57-49cf2199a46c-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:45.076914 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:45.076917    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwxmp\" (UniqueName: \"kubernetes.io/projected/f859f6e3-d493-48eb-9a57-49cf2199a46c-kube-api-access-mwxmp\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:45.095287 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:45.095259    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:06:45.643491 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:45.643465    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"
Oct 30 13:06:45.643650 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:45.643495    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw" event={"ID":"f859f6e3-d493-48eb-9a57-49cf2199a46c","Type":"ContainerDied","Data":"0435993e49e5a4ed0453758c9c537b1f7b044d8b2989c828910f8c00bd12122c"}
Oct 30 13:06:45.643650 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:45.643549    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0435993e49e5a4ed0453758c9c537b1f7b044d8b2989c828910f8c00bd12122c"
Oct 30 13:06:45.990971 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:45.990935    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice/crio-eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:06:46.165604 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.165538    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:06:46.292669 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.292633    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-envoy\") pod \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") "
Oct 30 13:06:46.292871 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.292700    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-credential-socket\") pod \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") "
Oct 30 13:06:46.292871 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.292810    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-data\") pod \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") "
Oct 30 13:06:46.292871 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.292849    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istiod-ca-cert\") pod \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") "
Oct 30 13:06:46.293048 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.292946    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-workload-socket\") pod \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") "
Oct 30 13:06:46.293048 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.292982    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-token\" (UniqueName: \"kubernetes.io/projected/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-token\") pod \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") "
Oct 30 13:06:46.293048 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.293025    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-podinfo\") pod \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") "
Oct 30 13:06:46.293206 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.293187    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c0a91a-fa80-4a17-9278-6bc43b6b9847-kube-api-access-5nhsb\" (UniqueName: \"kubernetes.io/projected/12c0a91a-fa80-4a17-9278-6bc43b6b9847-kube-api-access-5nhsb\") pod \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") "
Oct 30 13:06:46.293206 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.293188    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-workload-socket" pod "12c0a91a-fa80-4a17-9278-6bc43b6b9847" (UID: "12c0a91a-fa80-4a17-9278-6bc43b6b9847"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:46.293321 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.293262    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-workload-certs\") pod \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\" (UID: \"12c0a91a-fa80-4a17-9278-6bc43b6b9847\") "
Oct 30 13:06:46.293321 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.293304    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istiod-ca-cert" pod "12c0a91a-fa80-4a17-9278-6bc43b6b9847" (UID: "12c0a91a-fa80-4a17-9278-6bc43b6b9847"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:46.293481 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.293443    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-data" pod "12c0a91a-fa80-4a17-9278-6bc43b6b9847" (UID: "12c0a91a-fa80-4a17-9278-6bc43b6b9847"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:46.293821 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.293779    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-workload-certs" pod "12c0a91a-fa80-4a17-9278-6bc43b6b9847" (UID: "12c0a91a-fa80-4a17-9278-6bc43b6b9847"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:46.294294 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.294212    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:46.294294 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.294240    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:46.294294 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.294257    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:46.294294 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.294271    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:46.294669 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.294639    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-credential-socket" pod "12c0a91a-fa80-4a17-9278-6bc43b6b9847" (UID: "12c0a91a-fa80-4a17-9278-6bc43b6b9847"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:46.295223 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.295201    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-envoy" pod "12c0a91a-fa80-4a17-9278-6bc43b6b9847" (UID: "12c0a91a-fa80-4a17-9278-6bc43b6b9847"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:46.295613 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.295591    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-token" pod "12c0a91a-fa80-4a17-9278-6bc43b6b9847" (UID: "12c0a91a-fa80-4a17-9278-6bc43b6b9847"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:46.295833 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.295808    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-podinfo" pod "12c0a91a-fa80-4a17-9278-6bc43b6b9847" (UID: "12c0a91a-fa80-4a17-9278-6bc43b6b9847"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:06:46.296150 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.296129    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c0a91a-fa80-4a17-9278-6bc43b6b9847-kube-api-access-5nhsb" pod "12c0a91a-fa80-4a17-9278-6bc43b6b9847" (UID: "12c0a91a-fa80-4a17-9278-6bc43b6b9847"). InnerVolumeSpecName "kube-api-access-5nhsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:46.417879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.417705    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:46.417879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.417749    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:46.417879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.417768    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nhsb\" (UniqueName: \"kubernetes.io/projected/12c0a91a-fa80-4a17-9278-6bc43b6b9847-kube-api-access-5nhsb\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:46.417879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.417783    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:46.417879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.417797    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c0a91a-fa80-4a17-9278-6bc43b6b9847-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:46.449677 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.449643    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"]
Oct 30 13:06:46.661169 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.661131    5837 generic.go:356] "Generic (PLEG): container finished" podID="61c702c3-5d1c-44d9-b02e-1fa7e139ca9c" containerID="a49be8fc2aeb78b901f2ac9f4947b7612f09e20680cb493db39ae4a38145ea7a" exitCode=0
Oct 30 13:06:46.661359 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.661202    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm" event={"ID":"61c702c3-5d1c-44d9-b02e-1fa7e139ca9c","Type":"ContainerDied","Data":"a49be8fc2aeb78b901f2ac9f4947b7612f09e20680cb493db39ae4a38145ea7a"}
Oct 30 13:06:46.663527 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.663490    5837 generic.go:356] "Generic (PLEG): container finished" podID="1c4895db-c87e-45f5-9317-e2643409cff3" containerID="bb428992badd8568939c0307d90c4fa52f70eb08cc8b7541ba6531cbbd7c5963" exitCode=0
Oct 30 13:06:46.663652 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.663574    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9" event={"ID":"1c4895db-c87e-45f5-9317-e2643409cff3","Type":"ContainerDied","Data":"bb428992badd8568939c0307d90c4fa52f70eb08cc8b7541ba6531cbbd7c5963"}
Oct 30 13:06:46.663943 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.663918    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"
Oct 30 13:06:46.665991 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.665965    5837 generic.go:356] "Generic (PLEG): container finished" podID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerID="1e137026135c14ff1fa6c5a001f792310a8472789567563ac88b32a81ebb1362" exitCode=0
Oct 30 13:06:46.666135 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.666027    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"
Oct 30 13:06:46.666135 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.666050    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs" event={"ID":"12c0a91a-fa80-4a17-9278-6bc43b6b9847","Type":"ContainerDied","Data":"1e137026135c14ff1fa6c5a001f792310a8472789567563ac88b32a81ebb1362"}
Oct 30 13:06:46.666135 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.666104    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs" event={"ID":"12c0a91a-fa80-4a17-9278-6bc43b6b9847","Type":"ContainerDied","Data":"cad5c1820f85e5552c84fd2212ee1a081ce39758476ec6b6a3162390e550788a"}
Oct 30 13:06:46.666309 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.666137    5837 scope.go:122] "RemoveContainer" containerID="1e137026135c14ff1fa6c5a001f792310a8472789567563ac88b32a81ebb1362"
Oct 30 13:06:46.683997 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.683974    5837 scope.go:122] "RemoveContainer" containerID="90ed49c8e55f0938be4d90747c666e3f6cdd7c4f59c568ab77f7e87bf7efd192"
Oct 30 13:06:46.701525 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.701506    5837 scope.go:122] "RemoveContainer" containerID="1e137026135c14ff1fa6c5a001f792310a8472789567563ac88b32a81ebb1362"
Oct 30 13:06:46.701940 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:46.701837    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e137026135c14ff1fa6c5a001f792310a8472789567563ac88b32a81ebb1362\": container with ID starting with 1e137026135c14ff1fa6c5a001f792310a8472789567563ac88b32a81ebb1362 not found: ID does not exist" containerID="1e137026135c14ff1fa6c5a001f792310a8472789567563ac88b32a81ebb1362"
Oct 30 13:06:46.701940 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.701867    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e137026135c14ff1fa6c5a001f792310a8472789567563ac88b32a81ebb1362"} err="failed to get container status \"1e137026135c14ff1fa6c5a001f792310a8472789567563ac88b32a81ebb1362\": rpc error: code = NotFound desc = could not find container \"1e137026135c14ff1fa6c5a001f792310a8472789567563ac88b32a81ebb1362\": container with ID starting with 1e137026135c14ff1fa6c5a001f792310a8472789567563ac88b32a81ebb1362 not found: ID does not exist"
Oct 30 13:06:46.701940 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.701888    5837 scope.go:122] "RemoveContainer" containerID="90ed49c8e55f0938be4d90747c666e3f6cdd7c4f59c568ab77f7e87bf7efd192"
Oct 30 13:06:46.702169 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:46.702149    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ed49c8e55f0938be4d90747c666e3f6cdd7c4f59c568ab77f7e87bf7efd192\": container with ID starting with 90ed49c8e55f0938be4d90747c666e3f6cdd7c4f59c568ab77f7e87bf7efd192 not found: ID does not exist" containerID="90ed49c8e55f0938be4d90747c666e3f6cdd7c4f59c568ab77f7e87bf7efd192"
Oct 30 13:06:46.702212 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.702179    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ed49c8e55f0938be4d90747c666e3f6cdd7c4f59c568ab77f7e87bf7efd192"} err="failed to get container status \"90ed49c8e55f0938be4d90747c666e3f6cdd7c4f59c568ab77f7e87bf7efd192\": rpc error: code = NotFound desc = could not find container \"90ed49c8e55f0938be4d90747c666e3f6cdd7c4f59c568ab77f7e87bf7efd192\": container with ID starting with 90ed49c8e55f0938be4d90747c666e3f6cdd7c4f59c568ab77f7e87bf7efd192 not found: ID does not exist"
Oct 30 13:06:46.720161 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.720132    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"
Oct 30 13:06:46.749120 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.749079    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"]
Oct 30 13:06:46.756794 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.756762    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-666577-openshift-default-f887989df-gvxxs"]
Oct 30 13:06:46.823378 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.823345    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4895db-c87e-45f5-9317-e2643409cff3-kube-api-access-cxr95\" (UniqueName: \"kubernetes.io/projected/1c4895db-c87e-45f5-9317-e2643409cff3-kube-api-access-cxr95\") pod \"1c4895db-c87e-45f5-9317-e2643409cff3\" (UID: \"1c4895db-c87e-45f5-9317-e2643409cff3\") "
Oct 30 13:06:46.823581 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.823540    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4895db-c87e-45f5-9317-e2643409cff3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4895db-c87e-45f5-9317-e2643409cff3-plugins-config-volume\") pod \"1c4895db-c87e-45f5-9317-e2643409cff3\" (UID: \"1c4895db-c87e-45f5-9317-e2643409cff3\") "
Oct 30 13:06:46.823730 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.823711    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c702c3-5d1c-44d9-b02e-1fa7e139ca9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c702c3-5d1c-44d9-b02e-1fa7e139ca9c-plugins-config-volume\") pod \"61c702c3-5d1c-44d9-b02e-1fa7e139ca9c\" (UID: \"61c702c3-5d1c-44d9-b02e-1fa7e139ca9c\") "
Oct 30 13:06:46.823847 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.823828    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c702c3-5d1c-44d9-b02e-1fa7e139ca9c-kube-api-access-jrfjp\" (UniqueName: \"kubernetes.io/projected/61c702c3-5d1c-44d9-b02e-1fa7e139ca9c-kube-api-access-jrfjp\") pod \"61c702c3-5d1c-44d9-b02e-1fa7e139ca9c\" (UID: \"61c702c3-5d1c-44d9-b02e-1fa7e139ca9c\") "
Oct 30 13:06:46.825019 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.824901    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4895db-c87e-45f5-9317-e2643409cff3-plugins-config-volume" pod "1c4895db-c87e-45f5-9317-e2643409cff3" (UID: "1c4895db-c87e-45f5-9317-e2643409cff3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:46.825138 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.825030    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c702c3-5d1c-44d9-b02e-1fa7e139ca9c-plugins-config-volume" pod "61c702c3-5d1c-44d9-b02e-1fa7e139ca9c" (UID: "61c702c3-5d1c-44d9-b02e-1fa7e139ca9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:46.826774 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.826745    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4895db-c87e-45f5-9317-e2643409cff3-kube-api-access-cxr95" pod "1c4895db-c87e-45f5-9317-e2643409cff3" (UID: "1c4895db-c87e-45f5-9317-e2643409cff3"). InnerVolumeSpecName "kube-api-access-cxr95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:46.826898 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.826780    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c702c3-5d1c-44d9-b02e-1fa7e139ca9c-kube-api-access-jrfjp" pod "61c702c3-5d1c-44d9-b02e-1fa7e139ca9c" (UID: "61c702c3-5d1c-44d9-b02e-1fa7e139ca9c"). InnerVolumeSpecName "kube-api-access-jrfjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:46.925799 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.925761    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c702c3-5d1c-44d9-b02e-1fa7e139ca9c-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:46.925799 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.925793    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrfjp\" (UniqueName: \"kubernetes.io/projected/61c702c3-5d1c-44d9-b02e-1fa7e139ca9c-kube-api-access-jrfjp\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:46.925997 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.925807    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxr95\" (UniqueName: \"kubernetes.io/projected/1c4895db-c87e-45f5-9317-e2643409cff3-kube-api-access-cxr95\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:46.925997 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:46.925820    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4895db-c87e-45f5-9317-e2643409cff3-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:47.607537 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:47.607492    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" path="/var/lib/kubelet/pods/12c0a91a-fa80-4a17-9278-6bc43b6b9847/volumes"
Oct 30 13:06:47.677101 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:47.677067    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"
Oct 30 13:06:47.677101 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:47.677085    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm" event={"ID":"61c702c3-5d1c-44d9-b02e-1fa7e139ca9c","Type":"ContainerDied","Data":"3abe0a877ac203afa156eb9c5a9bd7e4e4354ee0ba81567bc9528f28c0bed5aa"}
Oct 30 13:06:47.677371 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:47.677153    5837 scope.go:122] "RemoveContainer" containerID="a49be8fc2aeb78b901f2ac9f4947b7612f09e20680cb493db39ae4a38145ea7a"
Oct 30 13:06:47.679103 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:47.679074    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9" event={"ID":"1c4895db-c87e-45f5-9317-e2643409cff3","Type":"ContainerDied","Data":"2da41c00a0926613afe680d5270b693b700e5847c7d472ebac1c08b531b4ae79"}
Oct 30 13:06:47.679228 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:47.679085    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"
Oct 30 13:06:47.687997 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:47.687980    5837 scope.go:122] "RemoveContainer" containerID="bb428992badd8568939c0307d90c4fa52f70eb08cc8b7541ba6531cbbd7c5963"
Oct 30 13:06:47.734073 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:47.734047    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"]
Oct 30 13:06:47.750107 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:47.750068    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c81f42-8487bcd959-b92h9"]
Oct 30 13:06:47.755423 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:06:47.755382    5837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice/crio-eac78b79320801344116dea5701f0a35221b77a9f4f56bfac7e0c38c14512104\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8284bf7_b488_4b53_8804_4636dc743903.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:06:47.773056 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:47.773035    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"]
Oct 30 13:06:47.777249 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:47.777226    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c81f42-7b97585bf6-29zvm"]
Oct 30 13:06:49.606657 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:49.606624    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4895db-c87e-45f5-9317-e2643409cff3" path="/var/lib/kubelet/pods/1c4895db-c87e-45f5-9317-e2643409cff3/volumes"
Oct 30 13:06:49.607068 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:49.607054    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c702c3-5d1c-44d9-b02e-1fa7e139ca9c" path="/var/lib/kubelet/pods/61c702c3-5d1c-44d9-b02e-1fa7e139ca9c/volumes"
Oct 30 13:06:51.950453 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:51.950430    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 13:06:51.952866 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:51.952845    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 13:06:52.373311 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:52.373278    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"]
Oct 30 13:06:52.402458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:52.402419    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"]
Oct 30 13:06:53.289863 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.289829    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"]
Oct 30 13:06:53.291340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.291077    5837 cpu_manager.go:414] "containerMap: removing container" podUID="f859f6e3-d493-48eb-9a57-49cf2199a46c" containerName="revoke-keys"
Oct 30 13:06:53.291340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.291099    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f859f6e3-d493-48eb-9a57-49cf2199a46c" containerName="revoke-keys"
Oct 30 13:06:53.291340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.291135    5837 cpu_manager.go:414] "containerMap: removing container" podUID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerName="istio-proxy"
Oct 30 13:06:53.291340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.291144    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerName="istio-proxy"
Oct 30 13:06:53.291340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.291172    5837 cpu_manager.go:414] "containerMap: removing container" podUID="1c4895db-c87e-45f5-9317-e2643409cff3" containerName="payload-pre-processing"
Oct 30 13:06:53.291340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.291179    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4895db-c87e-45f5-9317-e2643409cff3" containerName="payload-pre-processing"
Oct 30 13:06:53.291340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.291202    5837 cpu_manager.go:414] "containerMap: removing container" podUID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerName="istio-proxy"
Oct 30 13:06:53.291340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.291209    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerName="istio-proxy"
Oct 30 13:06:53.291340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.291218    5837 cpu_manager.go:414] "containerMap: removing container" podUID="61c702c3-5d1c-44d9-b02e-1fa7e139ca9c" containerName="payload-processing"
Oct 30 13:06:53.291340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.291225    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c702c3-5d1c-44d9-b02e-1fa7e139ca9c" containerName="payload-processing"
Oct 30 13:06:53.291980 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.291363    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4895db-c87e-45f5-9317-e2643409cff3" containerName="payload-pre-processing"
Oct 30 13:06:53.291980 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.291380    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerName="istio-proxy"
Oct 30 13:06:53.291980 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.291388    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c702c3-5d1c-44d9-b02e-1fa7e139ca9c" containerName="payload-processing"
Oct 30 13:06:53.291980 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.291426    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f859f6e3-d493-48eb-9a57-49cf2199a46c" containerName="revoke-keys"
Oct 30 13:06:53.316343 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.316316    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"]
Oct 30 13:06:53.316589 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.316445    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"
Oct 30 13:06:53.317722 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.317701    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c0a91a-fa80-4a17-9278-6bc43b6b9847" containerName="istio-proxy"
Oct 30 13:06:53.322020 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.322003    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"]
Oct 30 13:06:53.322178 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.322167    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"]
Oct 30 13:06:53.322344 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.322334    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"
Oct 30 13:06:53.337711 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.337691    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:06:53.338079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.338013    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:06:53.343068 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.343050    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:06:53.346545 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.346505    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:06:53.541915 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.541841    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfgjz\" (UniqueName: \"kubernetes.io/projected/15f36d2f-d742-4b02-a0f6-2ea63c70f471-kube-api-access-tfgjz\") pod \"maas-api-revoke-sub-7d9a8b3f-5ncq8\" (UID: \"15f36d2f-d742-4b02-a0f6-2ea63c70f471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"
Oct 30 13:06:53.542088 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.542067    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15f36d2f-d742-4b02-a0f6-2ea63c70f471-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d9a8b3f-5ncq8\" (UID: \"15f36d2f-d742-4b02-a0f6-2ea63c70f471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"
Oct 30 13:06:53.542135 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.542102    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15f36d2f-d742-4b02-a0f6-2ea63c70f471-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d9a8b3f-5ncq8\" (UID: \"15f36d2f-d742-4b02-a0f6-2ea63c70f471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"
Oct 30 13:06:53.542135 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.542121    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-727805cf-zbqr9\" (UID: \"6e75167c-1f70-44d3-b782-f6c7c4ff64d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"
Oct 30 13:06:53.542233 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.542212    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfvtp\" (UniqueName: \"kubernetes.io/projected/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-kube-api-access-pfvtp\") pod \"maas-api-revoke-sub-727805cf-zbqr9\" (UID: \"6e75167c-1f70-44d3-b782-f6c7c4ff64d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"
Oct 30 13:06:53.542270 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.542247    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-727805cf-zbqr9\" (UID: \"6e75167c-1f70-44d3-b782-f6c7c4ff64d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"
Oct 30 13:06:53.643314 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.643281    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15f36d2f-d742-4b02-a0f6-2ea63c70f471-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d9a8b3f-5ncq8\" (UID: \"15f36d2f-d742-4b02-a0f6-2ea63c70f471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"
Oct 30 13:06:53.643314 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.643318    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15f36d2f-d742-4b02-a0f6-2ea63c70f471-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d9a8b3f-5ncq8\" (UID: \"15f36d2f-d742-4b02-a0f6-2ea63c70f471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"
Oct 30 13:06:53.643645 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.643335    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-727805cf-zbqr9\" (UID: \"6e75167c-1f70-44d3-b782-f6c7c4ff64d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"
Oct 30 13:06:53.643645 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.643371    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfvtp\" (UniqueName: \"kubernetes.io/projected/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-kube-api-access-pfvtp\") pod \"maas-api-revoke-sub-727805cf-zbqr9\" (UID: \"6e75167c-1f70-44d3-b782-f6c7c4ff64d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"
Oct 30 13:06:53.643645 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.643433    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-727805cf-zbqr9\" (UID: \"6e75167c-1f70-44d3-b782-f6c7c4ff64d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"
Oct 30 13:06:53.643645 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.643537    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfgjz\" (UniqueName: \"kubernetes.io/projected/15f36d2f-d742-4b02-a0f6-2ea63c70f471-kube-api-access-tfgjz\") pod \"maas-api-revoke-sub-7d9a8b3f-5ncq8\" (UID: \"15f36d2f-d742-4b02-a0f6-2ea63c70f471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"
Oct 30 13:06:53.644130 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.644106    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15f36d2f-d742-4b02-a0f6-2ea63c70f471-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d9a8b3f-5ncq8\" (UID: \"15f36d2f-d742-4b02-a0f6-2ea63c70f471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"
Oct 30 13:06:53.644374 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.644126    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-727805cf-zbqr9\" (UID: \"6e75167c-1f70-44d3-b782-f6c7c4ff64d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"
Oct 30 13:06:53.645908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.645883    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-727805cf-zbqr9\" (UID: \"6e75167c-1f70-44d3-b782-f6c7c4ff64d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"
Oct 30 13:06:53.645908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.645900    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15f36d2f-d742-4b02-a0f6-2ea63c70f471-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d9a8b3f-5ncq8\" (UID: \"15f36d2f-d742-4b02-a0f6-2ea63c70f471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"
Oct 30 13:06:53.662114 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.662091    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfvtp\" (UniqueName: \"kubernetes.io/projected/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-kube-api-access-pfvtp\") pod \"maas-api-revoke-sub-727805cf-zbqr9\" (UID: \"6e75167c-1f70-44d3-b782-f6c7c4ff64d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"
Oct 30 13:06:53.662231 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.662213    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfgjz\" (UniqueName: \"kubernetes.io/projected/15f36d2f-d742-4b02-a0f6-2ea63c70f471-kube-api-access-tfgjz\") pod \"maas-api-revoke-sub-7d9a8b3f-5ncq8\" (UID: \"15f36d2f-d742-4b02-a0f6-2ea63c70f471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"
Oct 30 13:06:53.688483 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.688460    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"
Oct 30 13:06:53.741755 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.741719    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"
Oct 30 13:06:53.995622 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:53.995590    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"]
Oct 30 13:06:53.996596 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:06:53.996571    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f36d2f_d742_4b02_a0f6_2ea63c70f471.slice/crio-7277bb951b53f08bed36a43e9cf5446c086b0881f0a9b809006018c4aee620bc WatchSource:0}: Error finding container 7277bb951b53f08bed36a43e9cf5446c086b0881f0a9b809006018c4aee620bc: Status 404 returned error can't find the container with id 7277bb951b53f08bed36a43e9cf5446c086b0881f0a9b809006018c4aee620bc
Oct 30 13:06:54.106831 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:54.106802    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"]
Oct 30 13:06:54.107445 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:06:54.107420    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e75167c_1f70_44d3_b782_f6c7c4ff64d1.slice/crio-f41c550006881a6ae4fef9cd99a7cc25f37b67b3de394a9068712dbd6c48d878 WatchSource:0}: Error finding container f41c550006881a6ae4fef9cd99a7cc25f37b67b3de394a9068712dbd6c48d878: Status 404 returned error can't find the container with id f41c550006881a6ae4fef9cd99a7cc25f37b67b3de394a9068712dbd6c48d878
Oct 30 13:06:54.747449 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:54.747408    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9" event={"ID":"6e75167c-1f70-44d3-b782-f6c7c4ff64d1","Type":"ContainerStarted","Data":"86de79ba7e8b5507ca82b7b7ffcf1d5ea090c3c612c8a48adf20c370be5087f9"}
Oct 30 13:06:54.747915 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:54.747463    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9" event={"ID":"6e75167c-1f70-44d3-b782-f6c7c4ff64d1","Type":"ContainerStarted","Data":"f41c550006881a6ae4fef9cd99a7cc25f37b67b3de394a9068712dbd6c48d878"}
Oct 30 13:06:54.749326 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:54.749299    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8" event={"ID":"15f36d2f-d742-4b02-a0f6-2ea63c70f471","Type":"ContainerStarted","Data":"dad8a9f0c282a4a2ae3db332e04977b6c23521082935abc9e178f222f7e66e61"}
Oct 30 13:06:54.749448 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:54.749350    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8" event={"ID":"15f36d2f-d742-4b02-a0f6-2ea63c70f471","Type":"ContainerStarted","Data":"7277bb951b53f08bed36a43e9cf5446c086b0881f0a9b809006018c4aee620bc"}
Oct 30 13:06:54.786836 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:54.786785    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9" podStartSLOduration=1.786767615 podStartE2EDuration="1.786767615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:54.785635841 +0000 UTC m=+1803.309920060" watchObservedRunningTime="2026-10-30 13:06:54.786767615 +0000 UTC m=+1803.311051849"
Oct 30 13:06:54.824725 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:54.824678    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8" podStartSLOduration=1.824660666 podStartE2EDuration="1.824660666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:54.822753916 +0000 UTC m=+1803.347038158" watchObservedRunningTime="2026-10-30 13:06:54.824660666 +0000 UTC m=+1803.348944884"
Oct 30 13:06:55.761476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:55.761439    5837 generic.go:356] "Generic (PLEG): container finished" podID="6e75167c-1f70-44d3-b782-f6c7c4ff64d1" containerID="86de79ba7e8b5507ca82b7b7ffcf1d5ea090c3c612c8a48adf20c370be5087f9" exitCode=0
Oct 30 13:06:55.761476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:55.761475    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9" event={"ID":"6e75167c-1f70-44d3-b782-f6c7c4ff64d1","Type":"ContainerDied","Data":"86de79ba7e8b5507ca82b7b7ffcf1d5ea090c3c612c8a48adf20c370be5087f9"}
Oct 30 13:06:55.764269 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:55.764240    5837 generic.go:356] "Generic (PLEG): container finished" podID="15f36d2f-d742-4b02-a0f6-2ea63c70f471" containerID="dad8a9f0c282a4a2ae3db332e04977b6c23521082935abc9e178f222f7e66e61" exitCode=0
Oct 30 13:06:55.764449 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:55.764304    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8" event={"ID":"15f36d2f-d742-4b02-a0f6-2ea63c70f471","Type":"ContainerDied","Data":"dad8a9f0c282a4a2ae3db332e04977b6c23521082935abc9e178f222f7e66e61"}
Oct 30 13:06:56.678667 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:56.678625    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"]
Oct 30 13:06:56.685881 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:56.685632    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c81f42-x297j"]
Oct 30 13:06:57.009854 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.009823    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"
Oct 30 13:06:57.050063 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.050040    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"
Oct 30 13:06:57.157620 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.157590    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-maas-api-xks-ca\") pod \"6e75167c-1f70-44d3-b782-f6c7c4ff64d1\" (UID: \"6e75167c-1f70-44d3-b782-f6c7c4ff64d1\") "
Oct 30 13:06:57.157789 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.157703    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15f36d2f-d742-4b02-a0f6-2ea63c70f471-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15f36d2f-d742-4b02-a0f6-2ea63c70f471-maas-api-xks-ca\") pod \"15f36d2f-d742-4b02-a0f6-2ea63c70f471\" (UID: \"15f36d2f-d742-4b02-a0f6-2ea63c70f471\") "
Oct 30 13:06:57.157789 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.157740    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f36d2f-d742-4b02-a0f6-2ea63c70f471-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15f36d2f-d742-4b02-a0f6-2ea63c70f471-maas-api-openshift-ca\") pod \"15f36d2f-d742-4b02-a0f6-2ea63c70f471\" (UID: \"15f36d2f-d742-4b02-a0f6-2ea63c70f471\") "
Oct 30 13:06:57.157908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.157795    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-kube-api-access-pfvtp\" (UniqueName: \"kubernetes.io/projected/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-kube-api-access-pfvtp\") pod \"6e75167c-1f70-44d3-b782-f6c7c4ff64d1\" (UID: \"6e75167c-1f70-44d3-b782-f6c7c4ff64d1\") "
Oct 30 13:06:57.157908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.157826    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f36d2f-d742-4b02-a0f6-2ea63c70f471-kube-api-access-tfgjz\" (UniqueName: \"kubernetes.io/projected/15f36d2f-d742-4b02-a0f6-2ea63c70f471-kube-api-access-tfgjz\") pod \"15f36d2f-d742-4b02-a0f6-2ea63c70f471\" (UID: \"15f36d2f-d742-4b02-a0f6-2ea63c70f471\") "
Oct 30 13:06:57.157908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.157850    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-maas-api-openshift-ca\") pod \"6e75167c-1f70-44d3-b782-f6c7c4ff64d1\" (UID: \"6e75167c-1f70-44d3-b782-f6c7c4ff64d1\") "
Oct 30 13:06:57.158208 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.158169    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f36d2f-d742-4b02-a0f6-2ea63c70f471-maas-api-openshift-ca" pod "15f36d2f-d742-4b02-a0f6-2ea63c70f471" (UID: "15f36d2f-d742-4b02-a0f6-2ea63c70f471"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:57.158437 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.158414    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-maas-api-openshift-ca" pod "6e75167c-1f70-44d3-b782-f6c7c4ff64d1" (UID: "6e75167c-1f70-44d3-b782-f6c7c4ff64d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:57.158594 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.158578    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15f36d2f-d742-4b02-a0f6-2ea63c70f471-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:57.158673 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.158600    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:57.160197 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.160158    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f36d2f-d742-4b02-a0f6-2ea63c70f471-kube-api-access-tfgjz" pod "15f36d2f-d742-4b02-a0f6-2ea63c70f471" (UID: "15f36d2f-d742-4b02-a0f6-2ea63c70f471"). InnerVolumeSpecName "kube-api-access-tfgjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:57.160299 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.160197    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-kube-api-access-pfvtp" pod "6e75167c-1f70-44d3-b782-f6c7c4ff64d1" (UID: "6e75167c-1f70-44d3-b782-f6c7c4ff64d1"). InnerVolumeSpecName "kube-api-access-pfvtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:57.160476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.160454    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-maas-api-xks-ca" pod "6e75167c-1f70-44d3-b782-f6c7c4ff64d1" (UID: "6e75167c-1f70-44d3-b782-f6c7c4ff64d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:57.160476 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.160460    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15f36d2f-d742-4b02-a0f6-2ea63c70f471-maas-api-xks-ca" pod "15f36d2f-d742-4b02-a0f6-2ea63c70f471" (UID: "15f36d2f-d742-4b02-a0f6-2ea63c70f471"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:57.259854 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.259824    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15f36d2f-d742-4b02-a0f6-2ea63c70f471-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:57.259854 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.259853    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfvtp\" (UniqueName: \"kubernetes.io/projected/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-kube-api-access-pfvtp\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:57.260061 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.259863    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfgjz\" (UniqueName: \"kubernetes.io/projected/15f36d2f-d742-4b02-a0f6-2ea63c70f471-kube-api-access-tfgjz\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:57.260061 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.259871    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e75167c-1f70-44d3-b782-f6c7c4ff64d1-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:06:57.374107 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.374064    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx" podUID="e7f0fa26-8bb5-469d-9580-f460f4acc319" containerName="payload-processing" containerID="cri-o://084d36cb2e22ab5ef14859489ee505880e19f628f246096ba4f211804d281ae6" gracePeriod=25
Oct 30 13:06:57.403369 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.403290    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b" podUID="8e8be106-2a83-4465-9011-8298fff4f945" containerName="payload-pre-processing" containerID="cri-o://0240843b84f53d68c417ab7f1b01245319b0ddea3baca9fbdb3aa116ada10acf" gracePeriod=25
Oct 30 13:06:57.606152 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.606069    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ef5ae74-ccbc-4e61-b92c-72d35130cb10" path="/var/lib/kubelet/pods/0ef5ae74-ccbc-4e61-b92c-72d35130cb10/volumes"
Oct 30 13:06:57.768965 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.768936    5837 scope.go:122] "RemoveContainer" containerID="88593a4f0aee942a8bce7fe2e6df0e5886f430f531f586a89ec6a566a3f43a47"
Oct 30 13:06:57.781982 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.781726    5837 scope.go:122] "RemoveContainer" containerID="778afdf036135bfd2d3916c84d72553328e1bf4c4b86a16e045b44fa15d6ad30"
Oct 30 13:06:57.783042 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.783018    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8"
Oct 30 13:06:57.783126 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.783050    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d9a8b3f-5ncq8" event={"ID":"15f36d2f-d742-4b02-a0f6-2ea63c70f471","Type":"ContainerDied","Data":"7277bb951b53f08bed36a43e9cf5446c086b0881f0a9b809006018c4aee620bc"}
Oct 30 13:06:57.783126 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.783100    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7277bb951b53f08bed36a43e9cf5446c086b0881f0a9b809006018c4aee620bc"
Oct 30 13:06:57.785067 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.785032    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9" event={"ID":"6e75167c-1f70-44d3-b782-f6c7c4ff64d1","Type":"ContainerDied","Data":"f41c550006881a6ae4fef9cd99a7cc25f37b67b3de394a9068712dbd6c48d878"}
Oct 30 13:06:57.785198 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.785084    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f41c550006881a6ae4fef9cd99a7cc25f37b67b3de394a9068712dbd6c48d878"
Oct 30 13:06:57.785198 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.785042    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727805cf-zbqr9"
Oct 30 13:06:57.794624 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:06:57.794606    5837 scope.go:122] "RemoveContainer" containerID="c493db9af9d2cbcf9223d8d6e59f0671a34b02485fc6094c235afca851eed2bd"
Oct 30 13:07:01.478114 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.478070    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" containerID="cri-o://79e41aab6c1322aa55874f2859d71ec5936e224097a3c5d40f6f10a23be24ab3" gracePeriod=45
Oct 30 13:07:01.492616 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.492577    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": dial tcp 10.134.0.90:8000: connect: connection refused"
Oct 30 13:07:01.773282 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.773194    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 13:07:01.837191 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.834864    5837 generic.go:356] "Generic (PLEG): container finished" podID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerID="79e41aab6c1322aa55874f2859d71ec5936e224097a3c5d40f6f10a23be24ab3" exitCode=0
Oct 30 13:07:01.837191 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.835033    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" event={"ID":"9ab4a57e-f122-4801-8bee-3fc3f6d550f3","Type":"ContainerDied","Data":"79e41aab6c1322aa55874f2859d71ec5936e224097a3c5d40f6f10a23be24ab3"}
Oct 30 13:07:01.837191 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.835071    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj" event={"ID":"9ab4a57e-f122-4801-8bee-3fc3f6d550f3","Type":"ContainerDied","Data":"7b9e0ff355579b506d078b7dc86660e581118bfcd09b1af9589d8a256b93bc63"}
Oct 30 13:07:01.837191 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.835093    5837 scope.go:122] "RemoveContainer" containerID="79e41aab6c1322aa55874f2859d71ec5936e224097a3c5d40f6f10a23be24ab3"
Oct 30 13:07:01.837191 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.835249    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"
Oct 30 13:07:01.867210 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.867169    5837 scope.go:122] "RemoveContainer" containerID="79e41aab6c1322aa55874f2859d71ec5936e224097a3c5d40f6f10a23be24ab3"
Oct 30 13:07:01.869101 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:07:01.869023    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e41aab6c1322aa55874f2859d71ec5936e224097a3c5d40f6f10a23be24ab3\": container with ID starting with 79e41aab6c1322aa55874f2859d71ec5936e224097a3c5d40f6f10a23be24ab3 not found: ID does not exist" containerID="79e41aab6c1322aa55874f2859d71ec5936e224097a3c5d40f6f10a23be24ab3"
Oct 30 13:07:01.869238 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.869116    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e41aab6c1322aa55874f2859d71ec5936e224097a3c5d40f6f10a23be24ab3"} err="failed to get container status \"79e41aab6c1322aa55874f2859d71ec5936e224097a3c5d40f6f10a23be24ab3\": rpc error: code = NotFound desc = could not find container \"79e41aab6c1322aa55874f2859d71ec5936e224097a3c5d40f6f10a23be24ab3\": container with ID starting with 79e41aab6c1322aa55874f2859d71ec5936e224097a3c5d40f6f10a23be24ab3 not found: ID does not exist"
Oct 30 13:07:01.899023 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.898987    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b" podUID="8e8be106-2a83-4465-9011-8298fff4f945" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:07:01.899231 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.899074    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx" podUID="e7f0fa26-8bb5-469d-9580-f460f4acc319" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:07:01.933938 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.933896    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-home\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-home\") pod \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") "
Oct 30 13:07:01.934141 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.934044    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-kube-api-access-ngb5w\" (UniqueName: \"kubernetes.io/projected/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-kube-api-access-ngb5w\") pod \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") "
Oct 30 13:07:01.934205 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.934155    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-tls-certs\") pod \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") "
Oct 30 13:07:01.934267 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.934229    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-tmp-dir\") pod \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") "
Oct 30 13:07:01.934267 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.934225    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-home" pod "9ab4a57e-f122-4801-8bee-3fc3f6d550f3" (UID: "9ab4a57e-f122-4801-8bee-3fc3f6d550f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:01.934420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.934384    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-dshm\") pod \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") "
Oct 30 13:07:01.934504 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.934462    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-model-cache\") pod \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\" (UID: \"9ab4a57e-f122-4801-8bee-3fc3f6d550f3\") "
Oct 30 13:07:01.934835 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.934803    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-tmp-dir" pod "9ab4a57e-f122-4801-8bee-3fc3f6d550f3" (UID: "9ab4a57e-f122-4801-8bee-3fc3f6d550f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:01.935089 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.935064    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-model-cache" pod "9ab4a57e-f122-4801-8bee-3fc3f6d550f3" (UID: "9ab4a57e-f122-4801-8bee-3fc3f6d550f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:01.935179 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.935153    5837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-home\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:01.935179 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.935172    5837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-tmp-dir\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:01.947465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.947375    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-kube-api-access-ngb5w" pod "9ab4a57e-f122-4801-8bee-3fc3f6d550f3" (UID: "9ab4a57e-f122-4801-8bee-3fc3f6d550f3"). InnerVolumeSpecName "kube-api-access-ngb5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:01.950449 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.949495    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-dshm" pod "9ab4a57e-f122-4801-8bee-3fc3f6d550f3" (UID: "9ab4a57e-f122-4801-8bee-3fc3f6d550f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:01.950449 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:01.949786    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-tls-certs" pod "9ab4a57e-f122-4801-8bee-3fc3f6d550f3" (UID: "9ab4a57e-f122-4801-8bee-3fc3f6d550f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:07:02.037926 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:02.037306    5837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-tls-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:02.037926 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:02.037349    5837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-dshm\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:02.037926 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:02.037362    5837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-model-cache\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:02.037926 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:02.037377    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngb5w\" (UniqueName: \"kubernetes.io/projected/9ab4a57e-f122-4801-8bee-3fc3f6d550f3-kube-api-access-ngb5w\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:02.238024 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:02.237987    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"]
Oct 30 13:07:02.243872 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:02.243837    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4cd3fb/distinct-2-w0-4cd3fb-kserve-7554687646-v2lrj"]
Oct 30 13:07:03.660422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:03.651886    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" path="/var/lib/kubelet/pods/9ab4a57e-f122-4801-8bee-3fc3f6d550f3/volumes"
Oct 30 13:07:06.373582 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:06.373533    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:06.441532 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:06.413363    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"]
Oct 30 13:07:06.442097 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:06.442051    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w" podUID="bb740ce8-a013-43fd-b48c-96561ca1d3c5" containerName="istio-proxy" containerID="cri-o://1de89d1ceee28b8e7a110d747a147e27255c4b2f663c467cd741470dcf262460" gracePeriod=30
Oct 30 13:07:06.473487 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:06.473447    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"]
Oct 30 13:07:06.484437 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:06.473986    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp" podUID="88a518b4-df4c-4b56-b56a-a979acf3ca15" containerName="istio-proxy" containerID="cri-o://b9611a38f9ec36e924b2c3dc73dc6c4466b78b6da977651c581a6d627c86a616" gracePeriod=30
Oct 30 13:07:06.544890 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:06.544846    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"]
Oct 30 13:07:06.572422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:06.548683    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy" containerID="cri-o://2b672a2041c0b2733c3c8702d20c71f1d430f6f5a6b7a45dc87a080260b40172" gracePeriod=30
Oct 30 13:07:07.003635 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:07.002754    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:07:07.160541 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:07.160490    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 30 13:07:07.194535 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:07.194485    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": context deadline exceeded"
Oct 30 13:07:07.376594 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:07.376488    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:07.377950 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:07.377926    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:07:08.005719 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:08.005668    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:08.006609 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:08.006576    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:07:08.200293 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:08.200247    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:08.201135 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:08.201110    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 13:07:08.239616 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:08.239577    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:08.251424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:08.240441    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:07:08.377774 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:08.377677    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:08.618312 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:08.613154    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 13:07:09.016420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:09.006416    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:09.201420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:09.200003    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:07:09.241337 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:09.240645    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": context deadline exceeded"
Oct 30 13:07:09.388499 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:09.387699    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/ready\": context deadline exceeded"
Oct 30 13:07:09.428430 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:09.426755    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:07:09.439270 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:09.439242    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:07:11.313675 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.313642    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"]
Oct 30 13:07:11.322287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.322245    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4cd3fb-27bvw"]
Oct 30 13:07:11.606157 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.606124    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f859f6e3-d493-48eb-9a57-49cf2199a46c" path="/var/lib/kubelet/pods/f859f6e3-d493-48eb-9a57-49cf2199a46c/volumes"
Oct 30 13:07:11.666635 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.666602    5837 generic.go:356] "Generic (PLEG): container finished" podID="88a518b4-df4c-4b56-b56a-a979acf3ca15" containerID="b9611a38f9ec36e924b2c3dc73dc6c4466b78b6da977651c581a6d627c86a616" exitCode=0
Oct 30 13:07:11.666827 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.666759    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp" event={"ID":"88a518b4-df4c-4b56-b56a-a979acf3ca15","Type":"ContainerDied","Data":"b9611a38f9ec36e924b2c3dc73dc6c4466b78b6da977651c581a6d627c86a616"}
Oct 30 13:07:11.669295 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.669261    5837 generic.go:356] "Generic (PLEG): container finished" podID="bb740ce8-a013-43fd-b48c-96561ca1d3c5" containerID="1de89d1ceee28b8e7a110d747a147e27255c4b2f663c467cd741470dcf262460" exitCode=0
Oct 30 13:07:11.669458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.669325    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w" event={"ID":"bb740ce8-a013-43fd-b48c-96561ca1d3c5","Type":"ContainerDied","Data":"1de89d1ceee28b8e7a110d747a147e27255c4b2f663c467cd741470dcf262460"}
Oct 30 13:07:11.816304 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.816268    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 13:07:11.845279 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.845252    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 13:07:11.896877 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.896845    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx" podUID="e7f0fa26-8bb5-469d-9580-f460f4acc319" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:07:11.899002 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.898970    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b" podUID="8e8be106-2a83-4465-9011-8298fff4f945" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:07:11.912185 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.912158    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 13:07:11.955567 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.955530    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-workload-certs\") pod \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") "
Oct 30 13:07:11.955773 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.955578    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-workload-socket\") pod \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") "
Oct 30 13:07:11.955773 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.955637    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a518b4-df4c-4b56-b56a-a979acf3ca15-kube-api-access-dnhlk\" (UniqueName: \"kubernetes.io/projected/88a518b4-df4c-4b56-b56a-a979acf3ca15-kube-api-access-dnhlk\") pod \"88a518b4-df4c-4b56-b56a-a979acf3ca15\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") "
Oct 30 13:07:11.955773 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.955669    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-podinfo\") pod \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") "
Oct 30 13:07:11.955773 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.955704    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-data\") pod \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") "
Oct 30 13:07:11.955773 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.955736    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-podinfo\") pod \"88a518b4-df4c-4b56-b56a-a979acf3ca15\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") "
Oct 30 13:07:11.955773 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.955772    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-credential-socket\") pod \"88a518b4-df4c-4b56-b56a-a979acf3ca15\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") "
Oct 30 13:07:11.956287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.955812    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-data\") pod \"88a518b4-df4c-4b56-b56a-a979acf3ca15\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") "
Oct 30 13:07:11.956287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.955837    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a518b4-df4c-4b56-b56a-a979acf3ca15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a518b4-df4c-4b56-b56a-a979acf3ca15-istiod-ca-cert\") pod \"88a518b4-df4c-4b56-b56a-a979acf3ca15\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") "
Oct 30 13:07:11.956287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.955887    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-credential-socket\") pod \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") "
Oct 30 13:07:11.956287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.955886    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-workload-certs" pod "bb740ce8-a013-43fd-b48c-96561ca1d3c5" (UID: "bb740ce8-a013-43fd-b48c-96561ca1d3c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.956287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.955941    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-workload-socket\") pod \"88a518b4-df4c-4b56-b56a-a979acf3ca15\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") "
Oct 30 13:07:11.956287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.955976    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-token\" (UniqueName: \"kubernetes.io/projected/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-token\") pod \"88a518b4-df4c-4b56-b56a-a979acf3ca15\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") "
Oct 30 13:07:11.956287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.955998    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-workload-socket" pod "bb740ce8-a013-43fd-b48c-96561ca1d3c5" (UID: "bb740ce8-a013-43fd-b48c-96561ca1d3c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.956287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.956026    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-envoy\") pod \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") "
Oct 30 13:07:11.956287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.956126    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-credential-socket" pod "88a518b4-df4c-4b56-b56a-a979acf3ca15" (UID: "88a518b4-df4c-4b56-b56a-a979acf3ca15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.956287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.956138    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb740ce8-a013-43fd-b48c-96561ca1d3c5-kube-api-access-ncw6t\" (UniqueName: \"kubernetes.io/projected/bb740ce8-a013-43fd-b48c-96561ca1d3c5-kube-api-access-ncw6t\") pod \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") "
Oct 30 13:07:11.956287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.956185    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-token\" (UniqueName: \"kubernetes.io/projected/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-token\") pod \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") "
Oct 30 13:07:11.956287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.956224    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-credential-socket" pod "bb740ce8-a013-43fd-b48c-96561ca1d3c5" (UID: "bb740ce8-a013-43fd-b48c-96561ca1d3c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.956287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.956299    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-workload-certs\") pod \"88a518b4-df4c-4b56-b56a-a979acf3ca15\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") "
Oct 30 13:07:11.957329 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.956442    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istiod-ca-cert\") pod \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\" (UID: \"bb740ce8-a013-43fd-b48c-96561ca1d3c5\") "
Oct 30 13:07:11.957329 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.956447    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-data" pod "88a518b4-df4c-4b56-b56a-a979acf3ca15" (UID: "88a518b4-df4c-4b56-b56a-a979acf3ca15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.957329 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.956500    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-envoy\") pod \"88a518b4-df4c-4b56-b56a-a979acf3ca15\" (UID: \"88a518b4-df4c-4b56-b56a-a979acf3ca15\") "
Oct 30 13:07:11.957329 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.956668    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-data" pod "bb740ce8-a013-43fd-b48c-96561ca1d3c5" (UID: "bb740ce8-a013-43fd-b48c-96561ca1d3c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.957329 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.957039    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a518b4-df4c-4b56-b56a-a979acf3ca15-istiod-ca-cert" pod "88a518b4-df4c-4b56-b56a-a979acf3ca15" (UID: "88a518b4-df4c-4b56-b56a-a979acf3ca15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:11.957329 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.957290    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-workload-socket" pod "88a518b4-df4c-4b56-b56a-a979acf3ca15" (UID: "88a518b4-df4c-4b56-b56a-a979acf3ca15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.957689 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.957458    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.957689 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.957482    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.957689 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.957499    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.957689 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.957571    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a518b4-df4c-4b56-b56a-a979acf3ca15-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.957689 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.957587    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.957689 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.957603    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.957689 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.957617    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.957689 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.957630    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.958205 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.957926    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istiod-ca-cert" pod "bb740ce8-a013-43fd-b48c-96561ca1d3c5" (UID: "bb740ce8-a013-43fd-b48c-96561ca1d3c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:11.958975 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.958664    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a518b4-df4c-4b56-b56a-a979acf3ca15-kube-api-access-dnhlk" pod "88a518b4-df4c-4b56-b56a-a979acf3ca15" (UID: "88a518b4-df4c-4b56-b56a-a979acf3ca15"). InnerVolumeSpecName "kube-api-access-dnhlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:11.961026 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.959066    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-workload-certs" pod "88a518b4-df4c-4b56-b56a-a979acf3ca15" (UID: "88a518b4-df4c-4b56-b56a-a979acf3ca15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.961026 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.959272    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-token" pod "bb740ce8-a013-43fd-b48c-96561ca1d3c5" (UID: "bb740ce8-a013-43fd-b48c-96561ca1d3c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:11.961026 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.959464    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-podinfo" pod "bb740ce8-a013-43fd-b48c-96561ca1d3c5" (UID: "bb740ce8-a013-43fd-b48c-96561ca1d3c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:11.961456 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.961377    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-token" pod "88a518b4-df4c-4b56-b56a-a979acf3ca15" (UID: "88a518b4-df4c-4b56-b56a-a979acf3ca15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:11.961571 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.961466    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-envoy" pod "88a518b4-df4c-4b56-b56a-a979acf3ca15" (UID: "88a518b4-df4c-4b56-b56a-a979acf3ca15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.961571 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.961513    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-envoy" pod "bb740ce8-a013-43fd-b48c-96561ca1d3c5" (UID: "bb740ce8-a013-43fd-b48c-96561ca1d3c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.961814 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.961794    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb740ce8-a013-43fd-b48c-96561ca1d3c5-kube-api-access-ncw6t" pod "bb740ce8-a013-43fd-b48c-96561ca1d3c5" (UID: "bb740ce8-a013-43fd-b48c-96561ca1d3c5"). InnerVolumeSpecName "kube-api-access-ncw6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:11.962007 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:11.961984    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-podinfo" pod "88a518b4-df4c-4b56-b56a-a979acf3ca15" (UID: "88a518b4-df4c-4b56-b56a-a979acf3ca15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:12.058528 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.058494    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-workload-certs\") pod \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") "
Oct 30 13:07:12.058730 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.058538    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-credential-socket\") pod \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") "
Oct 30 13:07:12.058730 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.058594    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istiod-ca-cert\") pod \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") "
Oct 30 13:07:12.058730 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.058619    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-envoy\") pod \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") "
Oct 30 13:07:12.058730 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.058692    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-data\") pod \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") "
Oct 30 13:07:12.058958 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.058735    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-podinfo\") pod \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") "
Oct 30 13:07:12.058958 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.058818    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-kube-api-access-7d7lj\" (UniqueName: \"kubernetes.io/projected/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-kube-api-access-7d7lj\") pod \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") "
Oct 30 13:07:12.058958 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.058838    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-workload-certs" pod "bf6194a6-2f9c-4dc6-b517-8c819b850f3a" (UID: "bf6194a6-2f9c-4dc6-b517-8c819b850f3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:12.058958 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.058898    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-token\" (UniqueName: \"kubernetes.io/projected/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-token\") pod \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") "
Oct 30 13:07:12.058958 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.058905    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-credential-socket" pod "bf6194a6-2f9c-4dc6-b517-8c819b850f3a" (UID: "bf6194a6-2f9c-4dc6-b517-8c819b850f3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:12.058958 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.058947    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-workload-socket\") pod \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\" (UID: \"bf6194a6-2f9c-4dc6-b517-8c819b850f3a\") "
Oct 30 13:07:12.059311 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.059081    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-data" pod "bf6194a6-2f9c-4dc6-b517-8c819b850f3a" (UID: "bf6194a6-2f9c-4dc6-b517-8c819b850f3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:12.059311 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.059099    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istiod-ca-cert" pod "bf6194a6-2f9c-4dc6-b517-8c819b850f3a" (UID: "bf6194a6-2f9c-4dc6-b517-8c819b850f3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:12.059534 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.059370    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-workload-socket" pod "bf6194a6-2f9c-4dc6-b517-8c819b850f3a" (UID: "bf6194a6-2f9c-4dc6-b517-8c819b850f3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:12.060344 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.060320    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.060750 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.060358    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnhlk\" (UniqueName: \"kubernetes.io/projected/88a518b4-df4c-4b56-b56a-a979acf3ca15-kube-api-access-dnhlk\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.060750 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.060371    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.060750 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.060380    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.060750 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.060462    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.060750 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.060475    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.060750 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.060484    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.060750 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.060492    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncw6t\" (UniqueName: \"kubernetes.io/projected/bb740ce8-a013-43fd-b48c-96561ca1d3c5-kube-api-access-ncw6t\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.060750 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.060500    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.060750 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.060513    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.060750 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.060526    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb740ce8-a013-43fd-b48c-96561ca1d3c5-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.060750 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.060535    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.060750 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.060543    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a518b4-df4c-4b56-b56a-a979acf3ca15-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.060750 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.060551    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.060750 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.060559    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.062103 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.062080    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-token" pod "bf6194a6-2f9c-4dc6-b517-8c819b850f3a" (UID: "bf6194a6-2f9c-4dc6-b517-8c819b850f3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:12.062190 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.062149    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-kube-api-access-7d7lj" pod "bf6194a6-2f9c-4dc6-b517-8c819b850f3a" (UID: "bf6194a6-2f9c-4dc6-b517-8c819b850f3a"). InnerVolumeSpecName "kube-api-access-7d7lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:12.062348 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.062332    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-envoy" pod "bf6194a6-2f9c-4dc6-b517-8c819b850f3a" (UID: "bf6194a6-2f9c-4dc6-b517-8c819b850f3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:12.062466 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.062443    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-podinfo" pod "bf6194a6-2f9c-4dc6-b517-8c819b850f3a" (UID: "bf6194a6-2f9c-4dc6-b517-8c819b850f3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:12.162144 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.162070    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d7lj\" (UniqueName: \"kubernetes.io/projected/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-kube-api-access-7d7lj\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.162144 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.162098    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.162144 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.162107    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.162144 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.162116    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6194a6-2f9c-4dc6-b517-8c819b850f3a-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.679665 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.679632    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"
Oct 30 13:07:12.679665 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.679655    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w" event={"ID":"bb740ce8-a013-43fd-b48c-96561ca1d3c5","Type":"ContainerDied","Data":"38ece60583adb7eab2aa28a0e1126546c78fe0a53ac81e708db70290c123160a"}
Oct 30 13:07:12.680373 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.679724    5837 scope.go:122] "RemoveContainer" containerID="1de89d1ceee28b8e7a110d747a147e27255c4b2f663c467cd741470dcf262460"
Oct 30 13:07:12.682055 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.682012    5837 generic.go:356] "Generic (PLEG): container finished" podID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerID="2b672a2041c0b2733c3c8702d20c71f1d430f6f5a6b7a45dc87a080260b40172" exitCode=0
Oct 30 13:07:12.682282 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.682255    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" event={"ID":"bf6194a6-2f9c-4dc6-b517-8c819b850f3a","Type":"ContainerDied","Data":"2b672a2041c0b2733c3c8702d20c71f1d430f6f5a6b7a45dc87a080260b40172"}
Oct 30 13:07:12.682364 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.682332    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d" event={"ID":"bf6194a6-2f9c-4dc6-b517-8c819b850f3a","Type":"ContainerDied","Data":"7a910af4a34b2c2580b0691d1aef904bd6df5e4bcaea8de780afed6803feedc5"}
Oct 30 13:07:12.682488 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.682469    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"
Oct 30 13:07:12.685560 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.685536    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"
Oct 30 13:07:12.685739 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.685555    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp" event={"ID":"88a518b4-df4c-4b56-b56a-a979acf3ca15","Type":"ContainerDied","Data":"8c05c9990876b3c7f5d4f475f4721e2da9a746236dd742b40e32b2edc63515f8"}
Oct 30 13:07:12.694559 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.694532    5837 scope.go:122] "RemoveContainer" containerID="2b672a2041c0b2733c3c8702d20c71f1d430f6f5a6b7a45dc87a080260b40172"
Oct 30 13:07:12.706801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.706753    5837 scope.go:122] "RemoveContainer" containerID="2b672a2041c0b2733c3c8702d20c71f1d430f6f5a6b7a45dc87a080260b40172"
Oct 30 13:07:12.707200 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:07:12.707177    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b672a2041c0b2733c3c8702d20c71f1d430f6f5a6b7a45dc87a080260b40172\": container with ID starting with 2b672a2041c0b2733c3c8702d20c71f1d430f6f5a6b7a45dc87a080260b40172 not found: ID does not exist" containerID="2b672a2041c0b2733c3c8702d20c71f1d430f6f5a6b7a45dc87a080260b40172"
Oct 30 13:07:12.707316 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.707211    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b672a2041c0b2733c3c8702d20c71f1d430f6f5a6b7a45dc87a080260b40172"} err="failed to get container status \"2b672a2041c0b2733c3c8702d20c71f1d430f6f5a6b7a45dc87a080260b40172\": rpc error: code = NotFound desc = could not find container \"2b672a2041c0b2733c3c8702d20c71f1d430f6f5a6b7a45dc87a080260b40172\": container with ID starting with 2b672a2041c0b2733c3c8702d20c71f1d430f6f5a6b7a45dc87a080260b40172 not found: ID does not exist"
Oct 30 13:07:12.707316 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.707234    5837 scope.go:122] "RemoveContainer" containerID="b9611a38f9ec36e924b2c3dc73dc6c4466b78b6da977651c581a6d627c86a616"
Oct 30 13:07:12.747334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.747291    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"]
Oct 30 13:07:12.759656 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.759611    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-dc66w"]
Oct 30 13:07:12.801544 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.801505    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"]
Oct 30 13:07:12.805638 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.805595    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-zxl2d"]
Oct 30 13:07:12.838602 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.838496    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"]
Oct 30 13:07:12.842183 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:12.842145    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-sxndp"]
Oct 30 13:07:13.604952 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:13.604915    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a518b4-df4c-4b56-b56a-a979acf3ca15" path="/var/lib/kubelet/pods/88a518b4-df4c-4b56-b56a-a979acf3ca15/volumes"
Oct 30 13:07:13.605415 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:13.605377    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb740ce8-a013-43fd-b48c-96561ca1d3c5" path="/var/lib/kubelet/pods/bb740ce8-a013-43fd-b48c-96561ca1d3c5/volumes"
Oct 30 13:07:13.605851 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:13.605837    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" path="/var/lib/kubelet/pods/bf6194a6-2f9c-4dc6-b517-8c819b850f3a/volumes"
Oct 30 13:07:14.034033 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:14.033993    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"]
Oct 30 13:07:14.045201 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:14.045165    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf48c8b9-l7fz5"]
Oct 30 13:07:15.094988 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:07:15.094953    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:07:15.606029 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:15.605994    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f13e3f-c6ea-48b1-b60f-84418d3f0447" path="/var/lib/kubelet/pods/a2f13e3f-c6ea-48b1-b60f-84418d3f0447/volumes"
Oct 30 13:07:17.705179 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.705155    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"
Oct 30 13:07:17.725185 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.725158    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"
Oct 30 13:07:17.726601 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.726577    5837 generic.go:356] "Generic (PLEG): container finished" podID="8e8be106-2a83-4465-9011-8298fff4f945" containerID="0240843b84f53d68c417ab7f1b01245319b0ddea3baca9fbdb3aa116ada10acf" exitCode=0
Oct 30 13:07:17.726711 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.726656    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b" event={"ID":"8e8be106-2a83-4465-9011-8298fff4f945","Type":"ContainerDied","Data":"0240843b84f53d68c417ab7f1b01245319b0ddea3baca9fbdb3aa116ada10acf"}
Oct 30 13:07:17.726787 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.726713    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b" event={"ID":"8e8be106-2a83-4465-9011-8298fff4f945","Type":"ContainerDied","Data":"7179f7d09de49ecc4f63fb24eed1d6597d64e3ab6140b33a89ad77930ad06025"}
Oct 30 13:07:17.726787 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.726743    5837 scope.go:122] "RemoveContainer" containerID="0240843b84f53d68c417ab7f1b01245319b0ddea3baca9fbdb3aa116ada10acf"
Oct 30 13:07:17.728041 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.728021    5837 generic.go:356] "Generic (PLEG): container finished" podID="e7f0fa26-8bb5-469d-9580-f460f4acc319" containerID="084d36cb2e22ab5ef14859489ee505880e19f628f246096ba4f211804d281ae6" exitCode=0
Oct 30 13:07:17.728136 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.728058    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"
Oct 30 13:07:17.728136 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.728102    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx" event={"ID":"e7f0fa26-8bb5-469d-9580-f460f4acc319","Type":"ContainerDied","Data":"084d36cb2e22ab5ef14859489ee505880e19f628f246096ba4f211804d281ae6"}
Oct 30 13:07:17.728235 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.728170    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx" event={"ID":"e7f0fa26-8bb5-469d-9580-f460f4acc319","Type":"ContainerDied","Data":"2b5ad6565ed9ef99e939b423616b629d803df9237a32c158a695681fb4662cfb"}
Oct 30 13:07:17.737134 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.737117    5837 scope.go:122] "RemoveContainer" containerID="0240843b84f53d68c417ab7f1b01245319b0ddea3baca9fbdb3aa116ada10acf"
Oct 30 13:07:17.737510 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:07:17.737490    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0240843b84f53d68c417ab7f1b01245319b0ddea3baca9fbdb3aa116ada10acf\": container with ID starting with 0240843b84f53d68c417ab7f1b01245319b0ddea3baca9fbdb3aa116ada10acf not found: ID does not exist" containerID="0240843b84f53d68c417ab7f1b01245319b0ddea3baca9fbdb3aa116ada10acf"
Oct 30 13:07:17.737561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.737523    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0240843b84f53d68c417ab7f1b01245319b0ddea3baca9fbdb3aa116ada10acf"} err="failed to get container status \"0240843b84f53d68c417ab7f1b01245319b0ddea3baca9fbdb3aa116ada10acf\": rpc error: code = NotFound desc = could not find container \"0240843b84f53d68c417ab7f1b01245319b0ddea3baca9fbdb3aa116ada10acf\": container with ID starting with 0240843b84f53d68c417ab7f1b01245319b0ddea3baca9fbdb3aa116ada10acf not found: ID does not exist"
Oct 30 13:07:17.737561 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.737540    5837 scope.go:122] "RemoveContainer" containerID="084d36cb2e22ab5ef14859489ee505880e19f628f246096ba4f211804d281ae6"
Oct 30 13:07:17.738112 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.738084    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f0fa26-8bb5-469d-9580-f460f4acc319-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f0fa26-8bb5-469d-9580-f460f4acc319-plugins-config-volume\") pod \"e7f0fa26-8bb5-469d-9580-f460f4acc319\" (UID: \"e7f0fa26-8bb5-469d-9580-f460f4acc319\") "
Oct 30 13:07:17.738198 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.738123    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8be106-2a83-4465-9011-8298fff4f945-kube-api-access-mp48r\" (UniqueName: \"kubernetes.io/projected/8e8be106-2a83-4465-9011-8298fff4f945-kube-api-access-mp48r\") pod \"8e8be106-2a83-4465-9011-8298fff4f945\" (UID: \"8e8be106-2a83-4465-9011-8298fff4f945\") "
Oct 30 13:07:17.738266 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.738189    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e8be106-2a83-4465-9011-8298fff4f945-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8be106-2a83-4465-9011-8298fff4f945-plugins-config-volume\") pod \"8e8be106-2a83-4465-9011-8298fff4f945\" (UID: \"8e8be106-2a83-4465-9011-8298fff4f945\") "
Oct 30 13:07:17.738448 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.738298    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f0fa26-8bb5-469d-9580-f460f4acc319-kube-api-access-vwk4b\" (UniqueName: \"kubernetes.io/projected/e7f0fa26-8bb5-469d-9580-f460f4acc319-kube-api-access-vwk4b\") pod \"e7f0fa26-8bb5-469d-9580-f460f4acc319\" (UID: \"e7f0fa26-8bb5-469d-9580-f460f4acc319\") "
Oct 30 13:07:17.738576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.738548    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8be106-2a83-4465-9011-8298fff4f945-plugins-config-volume" pod "8e8be106-2a83-4465-9011-8298fff4f945" (UID: "8e8be106-2a83-4465-9011-8298fff4f945"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:17.738743 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.738720    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f0fa26-8bb5-469d-9580-f460f4acc319-plugins-config-volume" pod "e7f0fa26-8bb5-469d-9580-f460f4acc319" (UID: "e7f0fa26-8bb5-469d-9580-f460f4acc319"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:17.738977 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.738960    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f0fa26-8bb5-469d-9580-f460f4acc319-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:17.739046 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.738984    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8be106-2a83-4465-9011-8298fff4f945-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:17.740429 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.740410    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8be106-2a83-4465-9011-8298fff4f945-kube-api-access-mp48r" pod "8e8be106-2a83-4465-9011-8298fff4f945" (UID: "8e8be106-2a83-4465-9011-8298fff4f945"). InnerVolumeSpecName "kube-api-access-mp48r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:17.740946 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.740930    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f0fa26-8bb5-469d-9580-f460f4acc319-kube-api-access-vwk4b" pod "e7f0fa26-8bb5-469d-9580-f460f4acc319" (UID: "e7f0fa26-8bb5-469d-9580-f460f4acc319"). InnerVolumeSpecName "kube-api-access-vwk4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:17.749676 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.749651    5837 scope.go:122] "RemoveContainer" containerID="084d36cb2e22ab5ef14859489ee505880e19f628f246096ba4f211804d281ae6"
Oct 30 13:07:17.750017 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:07:17.749995    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084d36cb2e22ab5ef14859489ee505880e19f628f246096ba4f211804d281ae6\": container with ID starting with 084d36cb2e22ab5ef14859489ee505880e19f628f246096ba4f211804d281ae6 not found: ID does not exist" containerID="084d36cb2e22ab5ef14859489ee505880e19f628f246096ba4f211804d281ae6"
Oct 30 13:07:17.750113 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.750026    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084d36cb2e22ab5ef14859489ee505880e19f628f246096ba4f211804d281ae6"} err="failed to get container status \"084d36cb2e22ab5ef14859489ee505880e19f628f246096ba4f211804d281ae6\": rpc error: code = NotFound desc = could not find container \"084d36cb2e22ab5ef14859489ee505880e19f628f246096ba4f211804d281ae6\": container with ID starting with 084d36cb2e22ab5ef14859489ee505880e19f628f246096ba4f211804d281ae6 not found: ID does not exist"
Oct 30 13:07:17.840567 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.840531    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwk4b\" (UniqueName: \"kubernetes.io/projected/e7f0fa26-8bb5-469d-9580-f460f4acc319-kube-api-access-vwk4b\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:17.840567 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:17.840561    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp48r\" (UniqueName: \"kubernetes.io/projected/8e8be106-2a83-4465-9011-8298fff4f945-kube-api-access-mp48r\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:18.079210 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:18.079176    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"]
Oct 30 13:07:18.081934 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:18.081909    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4cd3fb-85db575d-t2nqx"]
Oct 30 13:07:18.291812 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:18.291726    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"]
Oct 30 13:07:18.292057 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:18.292023    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerName="istio-proxy" containerID="cri-o://c15f10603ebfeb5208d7085b75d9126d6eac6b670bb977f838fd42a48454499b" gracePeriod=30
Oct 30 13:07:18.314828 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:18.314794    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"]
Oct 30 13:07:18.315037 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:18.315014    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy" containerID="cri-o://89b4990e50ad129fbacdfe09be4bd082fe7b3f551b4c00b90e3732b19e5fe1e1" gracePeriod=30
Oct 30 13:07:18.319323 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:18.319294    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"]
Oct 30 13:07:18.319604 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:18.319577    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" podUID="48deaa90-e66d-454f-af97-1a5462424e82" containerName="istio-proxy" containerID="cri-o://64b7d7ab373443041363d3bb5d6152e8fc5c75532ba68ac8553786e55ba0ba42" gracePeriod=30
Oct 30 13:07:18.737591 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:18.737564    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"
Oct 30 13:07:18.824584 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:18.818595    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"]
Oct 30 13:07:18.827086 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:18.827055    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4cd3fb-6f6f98c44c-wzd8b"]
Oct 30 13:07:19.605509 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:19.605479    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8be106-2a83-4465-9011-8298fff4f945" path="/var/lib/kubelet/pods/8e8be106-2a83-4465-9011-8298fff4f945/volumes"
Oct 30 13:07:19.605909 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:19.605896    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f0fa26-8bb5-469d-9580-f460f4acc319" path="/var/lib/kubelet/pods/e7f0fa26-8bb5-469d-9580-f460f4acc319/volumes"
Oct 30 13:07:23.613005 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.612979    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 13:07:23.651410 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.651367    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 13:07:23.655630 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.655607    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 13:07:23.717955 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.717921    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982856f5-aa83-4e34-a199-ea238513db62-istio-token\" (UniqueName: \"kubernetes.io/projected/982856f5-aa83-4e34-a199-ea238513db62-istio-token\") pod \"982856f5-aa83-4e34-a199-ea238513db62\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") "
Oct 30 13:07:23.717955 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.717969    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-credential-socket\") pod \"982856f5-aa83-4e34-a199-ea238513db62\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") "
Oct 30 13:07:23.718220 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718049    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-workload-certs\") pod \"48deaa90-e66d-454f-af97-1a5462424e82\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") "
Oct 30 13:07:23.718220 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718087    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-credential-socket\") pod \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") "
Oct 30 13:07:23.718220 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718116    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48deaa90-e66d-454f-af97-1a5462424e82-kube-api-access-k87lf\" (UniqueName: \"kubernetes.io/projected/48deaa90-e66d-454f-af97-1a5462424e82-kube-api-access-k87lf\") pod \"48deaa90-e66d-454f-af97-1a5462424e82\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") "
Oct 30 13:07:23.718220 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718141    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-workload-socket\") pod \"982856f5-aa83-4e34-a199-ea238513db62\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") "
Oct 30 13:07:23.718220 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718168    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-istio-envoy\") pod \"48deaa90-e66d-454f-af97-1a5462424e82\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") "
Oct 30 13:07:23.718220 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718207    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/982856f5-aa83-4e34-a199-ea238513db62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982856f5-aa83-4e34-a199-ea238513db62-istiod-ca-cert\") pod \"982856f5-aa83-4e34-a199-ea238513db62\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") "
Oct 30 13:07:23.718632 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718235    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istiod-ca-cert\") pod \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") "
Oct 30 13:07:23.718632 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718269    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-workload-certs\") pod \"982856f5-aa83-4e34-a199-ea238513db62\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") "
Oct 30 13:07:23.718632 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718317    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-envoy\") pod \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") "
Oct 30 13:07:23.718632 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718356    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-workload-socket\") pod \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") "
Oct 30 13:07:23.718632 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718452    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-data\") pod \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") "
Oct 30 13:07:23.718632 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718453    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-workload-socket" pod "982856f5-aa83-4e34-a199-ea238513db62" (UID: "982856f5-aa83-4e34-a199-ea238513db62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.718632 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718495    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-istio-data\") pod \"982856f5-aa83-4e34-a199-ea238513db62\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") "
Oct 30 13:07:23.718632 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718527    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-istio-envoy\") pod \"982856f5-aa83-4e34-a199-ea238513db62\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") "
Oct 30 13:07:23.718632 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718570    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-kube-api-access-5cdt2\" (UniqueName: \"kubernetes.io/projected/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-kube-api-access-5cdt2\") pod \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") "
Oct 30 13:07:23.719079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718646    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/982856f5-aa83-4e34-a199-ea238513db62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982856f5-aa83-4e34-a199-ea238513db62-istio-podinfo\") pod \"982856f5-aa83-4e34-a199-ea238513db62\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") "
Oct 30 13:07:23.719079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718679    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-istio-data\") pod \"48deaa90-e66d-454f-af97-1a5462424e82\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") "
Oct 30 13:07:23.719079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718716    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48deaa90-e66d-454f-af97-1a5462424e82-istio-token\" (UniqueName: \"kubernetes.io/projected/48deaa90-e66d-454f-af97-1a5462424e82-istio-token\") pod \"48deaa90-e66d-454f-af97-1a5462424e82\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") "
Oct 30 13:07:23.719079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718779    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982856f5-aa83-4e34-a199-ea238513db62-kube-api-access-g2h4k\" (UniqueName: \"kubernetes.io/projected/982856f5-aa83-4e34-a199-ea238513db62-kube-api-access-g2h4k\") pod \"982856f5-aa83-4e34-a199-ea238513db62\" (UID: \"982856f5-aa83-4e34-a199-ea238513db62\") "
Oct 30 13:07:23.719079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718836    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-workload-socket\") pod \"48deaa90-e66d-454f-af97-1a5462424e82\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") "
Oct 30 13:07:23.719079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718873    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-token\" (UniqueName: \"kubernetes.io/projected/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-token\") pod \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") "
Oct 30 13:07:23.719079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718963    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-workload-certs\") pod \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") "
Oct 30 13:07:23.719079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718992    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48deaa90-e66d-454f-af97-1a5462424e82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48deaa90-e66d-454f-af97-1a5462424e82-istiod-ca-cert\") pod \"48deaa90-e66d-454f-af97-1a5462424e82\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") "
Oct 30 13:07:23.719079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.719027    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-podinfo\") pod \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\" (UID: \"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01\") "
Oct 30 13:07:23.719079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.719074    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48deaa90-e66d-454f-af97-1a5462424e82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48deaa90-e66d-454f-af97-1a5462424e82-istio-podinfo\") pod \"48deaa90-e66d-454f-af97-1a5462424e82\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") "
Oct 30 13:07:23.719652 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.719122    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-credential-socket\") pod \"48deaa90-e66d-454f-af97-1a5462424e82\" (UID: \"48deaa90-e66d-454f-af97-1a5462424e82\") "
Oct 30 13:07:23.719874 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.719849    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.721282 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718645    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-workload-certs" pod "48deaa90-e66d-454f-af97-1a5462424e82" (UID: "48deaa90-e66d-454f-af97-1a5462424e82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.721282 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.718822    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-credential-socket" pod "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" (UID: "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.721479 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.719619    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istiod-ca-cert" pod "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" (UID: "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:23.721479 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.719693    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-credential-socket" pod "982856f5-aa83-4e34-a199-ea238513db62" (UID: "982856f5-aa83-4e34-a199-ea238513db62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.721479 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.719969    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982856f5-aa83-4e34-a199-ea238513db62-istiod-ca-cert" pod "982856f5-aa83-4e34-a199-ea238513db62" (UID: "982856f5-aa83-4e34-a199-ea238513db62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:23.721479 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.720012    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48deaa90-e66d-454f-af97-1a5462424e82-istiod-ca-cert" pod "48deaa90-e66d-454f-af97-1a5462424e82" (UID: "48deaa90-e66d-454f-af97-1a5462424e82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:23.721479 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.720242    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-workload-socket" pod "48deaa90-e66d-454f-af97-1a5462424e82" (UID: "48deaa90-e66d-454f-af97-1a5462424e82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.721479 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.720992    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982856f5-aa83-4e34-a199-ea238513db62-istio-token" pod "982856f5-aa83-4e34-a199-ea238513db62" (UID: "982856f5-aa83-4e34-a199-ea238513db62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:23.721479 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.721255    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-istio-data" pod "48deaa90-e66d-454f-af97-1a5462424e82" (UID: "48deaa90-e66d-454f-af97-1a5462424e82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.722891 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.722864    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-credential-socket" pod "48deaa90-e66d-454f-af97-1a5462424e82" (UID: "48deaa90-e66d-454f-af97-1a5462424e82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.723109 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.723087    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-istio-data" pod "982856f5-aa83-4e34-a199-ea238513db62" (UID: "982856f5-aa83-4e34-a199-ea238513db62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.723183 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.723115    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-workload-socket" pod "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" (UID: "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.723291 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.723269    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-data" pod "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" (UID: "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.723532 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.723501    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-workload-certs" pod "982856f5-aa83-4e34-a199-ea238513db62" (UID: "982856f5-aa83-4e34-a199-ea238513db62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.723717 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.723695    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-workload-certs" pod "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" (UID: "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.723917 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.723888    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48deaa90-e66d-454f-af97-1a5462424e82-istio-podinfo" pod "48deaa90-e66d-454f-af97-1a5462424e82" (UID: "48deaa90-e66d-454f-af97-1a5462424e82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:23.725625 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.725599    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-istio-envoy" pod "982856f5-aa83-4e34-a199-ea238513db62" (UID: "982856f5-aa83-4e34-a199-ea238513db62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.725942 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.725849    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-istio-envoy" pod "48deaa90-e66d-454f-af97-1a5462424e82" (UID: "48deaa90-e66d-454f-af97-1a5462424e82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.725942 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.725894    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-token" pod "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" (UID: "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:23.725942 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.725919    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-podinfo" pod "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" (UID: "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:23.726260 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.726231    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-envoy" pod "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" (UID: "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.726484 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.726317    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48deaa90-e66d-454f-af97-1a5462424e82-kube-api-access-k87lf" pod "48deaa90-e66d-454f-af97-1a5462424e82" (UID: "48deaa90-e66d-454f-af97-1a5462424e82"). InnerVolumeSpecName "kube-api-access-k87lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:23.726553 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.726514    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982856f5-aa83-4e34-a199-ea238513db62-kube-api-access-g2h4k" pod "982856f5-aa83-4e34-a199-ea238513db62" (UID: "982856f5-aa83-4e34-a199-ea238513db62"). InnerVolumeSpecName "kube-api-access-g2h4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:23.726553 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.726533    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48deaa90-e66d-454f-af97-1a5462424e82-istio-token" pod "48deaa90-e66d-454f-af97-1a5462424e82" (UID: "48deaa90-e66d-454f-af97-1a5462424e82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:23.726553 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.726539    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-kube-api-access-5cdt2" pod "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" (UID: "2d7623e2-2633-4bb5-a6d9-f002c8ab8d01"). InnerVolumeSpecName "kube-api-access-5cdt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:23.727701 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.727676    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/982856f5-aa83-4e34-a199-ea238513db62-istio-podinfo" pod "982856f5-aa83-4e34-a199-ea238513db62" (UID: "982856f5-aa83-4e34-a199-ea238513db62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:23.751764 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.751729    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"]
Oct 30 13:07:23.752371 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752351    5837 cpu_manager.go:414] "containerMap: removing container" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy"
Oct 30 13:07:23.752371 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752369    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy"
Oct 30 13:07:23.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752442    5837 cpu_manager.go:414] "containerMap: removing container" podUID="48deaa90-e66d-454f-af97-1a5462424e82" containerName="istio-proxy"
Oct 30 13:07:23.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752449    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48deaa90-e66d-454f-af97-1a5462424e82" containerName="istio-proxy"
Oct 30 13:07:23.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752458    5837 cpu_manager.go:414] "containerMap: removing container" podUID="8e8be106-2a83-4465-9011-8298fff4f945" containerName="payload-pre-processing"
Oct 30 13:07:23.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752463    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8be106-2a83-4465-9011-8298fff4f945" containerName="payload-pre-processing"
Oct 30 13:07:23.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752476    5837 cpu_manager.go:414] "containerMap: removing container" podUID="e7f0fa26-8bb5-469d-9580-f460f4acc319" containerName="payload-processing"
Oct 30 13:07:23.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752481    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f0fa26-8bb5-469d-9580-f460f4acc319" containerName="payload-processing"
Oct 30 13:07:23.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752495    5837 cpu_manager.go:414] "containerMap: removing container" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main"
Oct 30 13:07:23.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752501    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main"
Oct 30 13:07:23.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752514    5837 cpu_manager.go:414] "containerMap: removing container" podUID="15f36d2f-d742-4b02-a0f6-2ea63c70f471" containerName="revoke-keys"
Oct 30 13:07:23.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752521    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f36d2f-d742-4b02-a0f6-2ea63c70f471" containerName="revoke-keys"
Oct 30 13:07:23.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752529    5837 cpu_manager.go:414] "containerMap: removing container" podUID="bb740ce8-a013-43fd-b48c-96561ca1d3c5" containerName="istio-proxy"
Oct 30 13:07:23.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752533    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb740ce8-a013-43fd-b48c-96561ca1d3c5" containerName="istio-proxy"
Oct 30 13:07:23.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752550    5837 cpu_manager.go:414] "containerMap: removing container" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerName="istio-proxy"
Oct 30 13:07:23.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752559    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerName="istio-proxy"
Oct 30 13:07:23.752570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752578    5837 cpu_manager.go:414] "containerMap: removing container" podUID="88a518b4-df4c-4b56-b56a-a979acf3ca15" containerName="istio-proxy"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752585    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a518b4-df4c-4b56-b56a-a979acf3ca15" containerName="istio-proxy"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752598    5837 cpu_manager.go:414] "containerMap: removing container" podUID="6e75167c-1f70-44d3-b782-f6c7c4ff64d1" containerName="revoke-keys"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752604    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e75167c-1f70-44d3-b782-f6c7c4ff64d1" containerName="revoke-keys"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752615    5837 cpu_manager.go:414] "containerMap: removing container" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752622    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752736    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6194a6-2f9c-4dc6-b517-8c819b850f3a" containerName="istio-proxy"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752774    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="48deaa90-e66d-454f-af97-1a5462424e82" containerName="istio-proxy"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752784    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8be106-2a83-4465-9011-8298fff4f945" containerName="payload-pre-processing"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752795    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f0fa26-8bb5-469d-9580-f460f4acc319" containerName="payload-processing"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752807    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab4a57e-f122-4801-8bee-3fc3f6d550f3" containerName="main"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752816    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f36d2f-d742-4b02-a0f6-2ea63c70f471" containerName="revoke-keys"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752824    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb740ce8-a013-43fd-b48c-96561ca1d3c5" containerName="istio-proxy"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752835    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerName="istio-proxy"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752843    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a518b4-df4c-4b56-b56a-a979acf3ca15" containerName="istio-proxy"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752852    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e75167c-1f70-44d3-b782-f6c7c4ff64d1" containerName="revoke-keys"
Oct 30 13:07:23.753801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.752862    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="982856f5-aa83-4e34-a199-ea238513db62" containerName="istio-proxy"
Oct 30 13:07:23.764580 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.764550    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.791697 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.791022    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-026c2f-openshift-default-dockercfg-kcwm7\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.839636    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-data\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.839702    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-workload-certs\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.839759    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-token\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.839790    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-envoy\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.839824    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd62k\" (UniqueName: \"kubernetes.io/projected/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-kube-api-access-pd62k\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.839851    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-workload-socket\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.839967    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istiod-ca-cert\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840004    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-credential-socket\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840078    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-podinfo\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840150    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48deaa90-e66d-454f-af97-1a5462424e82-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840166    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2h4k\" (UniqueName: \"kubernetes.io/projected/982856f5-aa83-4e34-a199-ea238513db62-kube-api-access-g2h4k\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840180    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840194    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840208    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840222    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48deaa90-e66d-454f-af97-1a5462424e82-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840234    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840247    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48deaa90-e66d-454f-af97-1a5462424e82-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840260    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840273    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982856f5-aa83-4e34-a199-ea238513db62-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840286    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840299    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840312    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840326    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k87lf\" (UniqueName: \"kubernetes.io/projected/48deaa90-e66d-454f-af97-1a5462424e82-kube-api-access-k87lf\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840339    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840354    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982856f5-aa83-4e34-a199-ea238513db62-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840366    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.841882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.840379    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.852187 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.850726    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"]
Oct 30 13:07:23.853334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.853191    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.853334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.853216    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.853334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.853227    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.853334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.853237    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.853334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.853245    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982856f5-aa83-4e34-a199-ea238513db62-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.853334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.853253    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cdt2\" (UniqueName: \"kubernetes.io/projected/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01-kube-api-access-5cdt2\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.853334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.853261    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982856f5-aa83-4e34-a199-ea238513db62-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.853334 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.853270    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48deaa90-e66d-454f-af97-1a5462424e82-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.856310 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.855676    5837 generic.go:356] "Generic (PLEG): container finished" podID="982856f5-aa83-4e34-a199-ea238513db62" containerID="89b4990e50ad129fbacdfe09be4bd082fe7b3f551b4c00b90e3732b19e5fe1e1" exitCode=0
Oct 30 13:07:23.856310 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.855749    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"
Oct 30 13:07:23.856310 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.855764    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" event={"ID":"982856f5-aa83-4e34-a199-ea238513db62","Type":"ContainerDied","Data":"89b4990e50ad129fbacdfe09be4bd082fe7b3f551b4c00b90e3732b19e5fe1e1"}
Oct 30 13:07:23.856310 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.855886    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt" event={"ID":"982856f5-aa83-4e34-a199-ea238513db62","Type":"ContainerDied","Data":"f53081db23b153261f511ab4f6c0b86ed80b9342a5cb72789a2a787267d6e741"}
Oct 30 13:07:23.856310 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.855915    5837 scope.go:122] "RemoveContainer" containerID="89b4990e50ad129fbacdfe09be4bd082fe7b3f551b4c00b90e3732b19e5fe1e1"
Oct 30 13:07:23.859789 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.859433    5837 generic.go:356] "Generic (PLEG): container finished" podID="48deaa90-e66d-454f-af97-1a5462424e82" containerID="64b7d7ab373443041363d3bb5d6152e8fc5c75532ba68ac8553786e55ba0ba42" exitCode=0
Oct 30 13:07:23.859789 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.859542    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" event={"ID":"48deaa90-e66d-454f-af97-1a5462424e82","Type":"ContainerDied","Data":"64b7d7ab373443041363d3bb5d6152e8fc5c75532ba68ac8553786e55ba0ba42"}
Oct 30 13:07:23.859789 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.859571    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554" event={"ID":"48deaa90-e66d-454f-af97-1a5462424e82","Type":"ContainerDied","Data":"599510a4c8a88568ea61f5d601f8df8a06c2505ddcdee4d42720b21af871c108"}
Oct 30 13:07:23.859789 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.859629    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"
Oct 30 13:07:23.868069 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.865478    5837 generic.go:356] "Generic (PLEG): container finished" podID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" containerID="c15f10603ebfeb5208d7085b75d9126d6eac6b670bb977f838fd42a48454499b" exitCode=0
Oct 30 13:07:23.868069 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.865541    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"
Oct 30 13:07:23.868069 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.865623    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" event={"ID":"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01","Type":"ContainerDied","Data":"c15f10603ebfeb5208d7085b75d9126d6eac6b670bb977f838fd42a48454499b"}
Oct 30 13:07:23.868069 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.865665    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs" event={"ID":"2d7623e2-2633-4bb5-a6d9-f002c8ab8d01","Type":"ContainerDied","Data":"eafff79a8ac2a725cc753facbb3af0bf85b83b2441fc52e399297d3836d24c78"}
Oct 30 13:07:23.872785 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.872678    5837 scope.go:122] "RemoveContainer" containerID="89b4990e50ad129fbacdfe09be4bd082fe7b3f551b4c00b90e3732b19e5fe1e1"
Oct 30 13:07:23.872990 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:07:23.872971    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b4990e50ad129fbacdfe09be4bd082fe7b3f551b4c00b90e3732b19e5fe1e1\": container with ID starting with 89b4990e50ad129fbacdfe09be4bd082fe7b3f551b4c00b90e3732b19e5fe1e1 not found: ID does not exist" containerID="89b4990e50ad129fbacdfe09be4bd082fe7b3f551b4c00b90e3732b19e5fe1e1"
Oct 30 13:07:23.873171 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.872998    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b4990e50ad129fbacdfe09be4bd082fe7b3f551b4c00b90e3732b19e5fe1e1"} err="failed to get container status \"89b4990e50ad129fbacdfe09be4bd082fe7b3f551b4c00b90e3732b19e5fe1e1\": rpc error: code = NotFound desc = could not find container \"89b4990e50ad129fbacdfe09be4bd082fe7b3f551b4c00b90e3732b19e5fe1e1\": container with ID starting with 89b4990e50ad129fbacdfe09be4bd082fe7b3f551b4c00b90e3732b19e5fe1e1 not found: ID does not exist"
Oct 30 13:07:23.873171 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.873014    5837 scope.go:122] "RemoveContainer" containerID="64b7d7ab373443041363d3bb5d6152e8fc5c75532ba68ac8553786e55ba0ba42"
Oct 30 13:07:23.888505 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.888481    5837 scope.go:122] "RemoveContainer" containerID="64b7d7ab373443041363d3bb5d6152e8fc5c75532ba68ac8553786e55ba0ba42"
Oct 30 13:07:23.888846 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:07:23.888819    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b7d7ab373443041363d3bb5d6152e8fc5c75532ba68ac8553786e55ba0ba42\": container with ID starting with 64b7d7ab373443041363d3bb5d6152e8fc5c75532ba68ac8553786e55ba0ba42 not found: ID does not exist" containerID="64b7d7ab373443041363d3bb5d6152e8fc5c75532ba68ac8553786e55ba0ba42"
Oct 30 13:07:23.888940 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.888857    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b7d7ab373443041363d3bb5d6152e8fc5c75532ba68ac8553786e55ba0ba42"} err="failed to get container status \"64b7d7ab373443041363d3bb5d6152e8fc5c75532ba68ac8553786e55ba0ba42\": rpc error: code = NotFound desc = could not find container \"64b7d7ab373443041363d3bb5d6152e8fc5c75532ba68ac8553786e55ba0ba42\": container with ID starting with 64b7d7ab373443041363d3bb5d6152e8fc5c75532ba68ac8553786e55ba0ba42 not found: ID does not exist"
Oct 30 13:07:23.888940 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.888882    5837 scope.go:122] "RemoveContainer" containerID="c15f10603ebfeb5208d7085b75d9126d6eac6b670bb977f838fd42a48454499b"
Oct 30 13:07:23.900719 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.900701    5837 scope.go:122] "RemoveContainer" containerID="c15f10603ebfeb5208d7085b75d9126d6eac6b670bb977f838fd42a48454499b"
Oct 30 13:07:23.901010 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:07:23.900988    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15f10603ebfeb5208d7085b75d9126d6eac6b670bb977f838fd42a48454499b\": container with ID starting with c15f10603ebfeb5208d7085b75d9126d6eac6b670bb977f838fd42a48454499b not found: ID does not exist" containerID="c15f10603ebfeb5208d7085b75d9126d6eac6b670bb977f838fd42a48454499b"
Oct 30 13:07:23.901111 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.901015    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15f10603ebfeb5208d7085b75d9126d6eac6b670bb977f838fd42a48454499b"} err="failed to get container status \"c15f10603ebfeb5208d7085b75d9126d6eac6b670bb977f838fd42a48454499b\": rpc error: code = NotFound desc = could not find container \"c15f10603ebfeb5208d7085b75d9126d6eac6b670bb977f838fd42a48454499b\": container with ID starting with c15f10603ebfeb5208d7085b75d9126d6eac6b670bb977f838fd42a48454499b not found: ID does not exist"
Oct 30 13:07:23.936341 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.936316    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"]
Oct 30 13:07:23.942854 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.942820    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-5lpwt"]
Oct 30 13:07:23.953942 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.953922    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-podinfo\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.954041 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.953974    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-data\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.954041 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.954025    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-workload-certs\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.954175 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.954092    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-token\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.954175 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.954124    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-envoy\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.954175 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.954162    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd62k\" (UniqueName: \"kubernetes.io/projected/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-kube-api-access-pd62k\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.954318 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.954189    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-workload-socket\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.954371 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.954322    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istiod-ca-cert\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.954371 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.954362    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-credential-socket\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.954503 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.954474    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-data\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.954560 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.954511    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-workload-certs\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.954728 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.954705    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-workload-socket\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.955011 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.954830    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-credential-socket\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.955388 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.955364    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istiod-ca-cert\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.956756 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.956729    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-envoy\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.956756 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.956749    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-podinfo\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.973191 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.973169    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-token\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.976581 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.976562    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"]
Oct 30 13:07:23.976654 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.976622    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd62k\" (UniqueName: \"kubernetes.io/projected/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-kube-api-access-pd62k\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:23.982624 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:23.982606    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-nx554"]
Oct 30 13:07:24.029002 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:24.028976    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"]
Oct 30 13:07:24.031790 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:24.031748    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-g6dzs"]
Oct 30 13:07:24.166712 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:24.166591    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:24.487790 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:24.487764    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"]
Oct 30 13:07:24.489914 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:07:24.489860    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8e4b82_23e0_4e40_a20a_1ec91fba2aa5.slice/crio-7779493986c964a1d0d90f3d43cb5842718fbd7076dfae0935b8814f720ddabb WatchSource:0}: Error finding container 7779493986c964a1d0d90f3d43cb5842718fbd7076dfae0935b8814f720ddabb: Status 404 returned error can't find the container with id 7779493986c964a1d0d90f3d43cb5842718fbd7076dfae0935b8814f720ddabb
Oct 30 13:07:24.493108 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:24.493070    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:07:24.493255 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:24.493150    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:07:24.493255 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:24.493197    5837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:07:24.878372 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:24.878273    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl" event={"ID":"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5","Type":"ContainerStarted","Data":"46625b6a15e0aa6951c1cacf17549253710ea07957122bf593aadbac350d8584"}
Oct 30 13:07:24.878372 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:24.878333    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl" event={"ID":"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5","Type":"ContainerStarted","Data":"7779493986c964a1d0d90f3d43cb5842718fbd7076dfae0935b8814f720ddabb"}
Oct 30 13:07:24.939920 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:24.939854    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl" podStartSLOduration=1.939837963 podStartE2EDuration="1.939837963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:24.937325729 +0000 UTC m=+1833.461609948" watchObservedRunningTime="2026-10-30 13:07:24.939837963 +0000 UTC m=+1833.464122182"
Oct 30 13:07:25.167142 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:25.167052    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:25.168809 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:25.168775    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:07:25.168957 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:25.168829    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl" podUID="eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 13:07:25.615574 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:25.615540    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d7623e2-2633-4bb5-a6d9-f002c8ab8d01" path="/var/lib/kubelet/pods/2d7623e2-2633-4bb5-a6d9-f002c8ab8d01/volumes"
Oct 30 13:07:25.616024 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:25.616010    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48deaa90-e66d-454f-af97-1a5462424e82" path="/var/lib/kubelet/pods/48deaa90-e66d-454f-af97-1a5462424e82/volumes"
Oct 30 13:07:25.616456 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:25.616442    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982856f5-aa83-4e34-a199-ea238513db62" path="/var/lib/kubelet/pods/982856f5-aa83-4e34-a199-ea238513db62/volumes"
Oct 30 13:07:26.167688 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:26.167648    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:07:26.168209 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:26.167716    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl" podUID="eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 13:07:27.167381 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:27.167338    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:07:27.167600 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:27.167421    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl" podUID="eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 13:07:28.167261 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:28.167230    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:07:28.167652 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:28.167278    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl" podUID="eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 13:07:29.167500 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:29.167470    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:07:29.167901 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:29.167527    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl" podUID="eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 13:07:30.167994 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:30.167954    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:07:30.168448 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:30.168022    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl" podUID="eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 13:07:31.167814 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:31.167772    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:07:31.167979 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:31.167846    5837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl" podUID="eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 13:07:32.171716 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:32.171686    5837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:32.172198 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:32.171993    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:32.172917 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:32.172898    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:07:45.094201 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:07:45.094159    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:07:56.225558 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.225526    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"]
Oct 30 13:07:56.229670 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.229651    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"
Oct 30 13:07:56.245075 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.245016    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:07:56.256654 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.256630    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:07:56.256802 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.256772    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:07:56.256802 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.256794    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:07:56.261442 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.261418    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"]
Oct 30 13:07:56.330020 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.329990    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggt4c\" (UniqueName: \"kubernetes.io/projected/6af65555-7415-4c48-beba-44e6ada79954-kube-api-access-ggt4c\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt\" (UID: \"6af65555-7415-4c48-beba-44e6ada79954\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"
Oct 30 13:07:56.330180 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.330059    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af65555-7415-4c48-beba-44e6ada79954-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt\" (UID: \"6af65555-7415-4c48-beba-44e6ada79954\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"
Oct 30 13:07:56.330180 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.330133    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af65555-7415-4c48-beba-44e6ada79954-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt\" (UID: \"6af65555-7415-4c48-beba-44e6ada79954\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"
Oct 30 13:07:56.431459 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.431386    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggt4c\" (UniqueName: \"kubernetes.io/projected/6af65555-7415-4c48-beba-44e6ada79954-kube-api-access-ggt4c\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt\" (UID: \"6af65555-7415-4c48-beba-44e6ada79954\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"
Oct 30 13:07:56.431618 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.431533    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af65555-7415-4c48-beba-44e6ada79954-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt\" (UID: \"6af65555-7415-4c48-beba-44e6ada79954\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"
Oct 30 13:07:56.431618 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.431578    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af65555-7415-4c48-beba-44e6ada79954-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt\" (UID: \"6af65555-7415-4c48-beba-44e6ada79954\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"
Oct 30 13:07:56.432242 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.432222    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af65555-7415-4c48-beba-44e6ada79954-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt\" (UID: \"6af65555-7415-4c48-beba-44e6ada79954\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"
Oct 30 13:07:56.433938 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.433921    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af65555-7415-4c48-beba-44e6ada79954-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt\" (UID: \"6af65555-7415-4c48-beba-44e6ada79954\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"
Oct 30 13:07:56.450448 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.450420    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggt4c\" (UniqueName: \"kubernetes.io/projected/6af65555-7415-4c48-beba-44e6ada79954-kube-api-access-ggt4c\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt\" (UID: \"6af65555-7415-4c48-beba-44e6ada79954\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"
Oct 30 13:07:56.555049 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.555019    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"
Oct 30 13:07:56.851443 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:56.851416    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"]
Oct 30 13:07:56.852866 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:07:56.852831    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af65555_7415_4c48_beba_44e6ada79954.slice/crio-301d54b8aa1df9b232c4049abe9cc49ddce7fdd499af099eb58311d8d7409a0b WatchSource:0}: Error finding container 301d54b8aa1df9b232c4049abe9cc49ddce7fdd499af099eb58311d8d7409a0b: Status 404 returned error can't find the container with id 301d54b8aa1df9b232c4049abe9cc49ddce7fdd499af099eb58311d8d7409a0b
Oct 30 13:07:57.213010 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:57.212973    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt" event={"ID":"6af65555-7415-4c48-beba-44e6ada79954","Type":"ContainerStarted","Data":"34079e242579d2d335185bf89daafa4a1a89dbbd88deab3364ea9133745abd42"}
Oct 30 13:07:57.213133 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:57.213031    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt" event={"ID":"6af65555-7415-4c48-beba-44e6ada79954","Type":"ContainerStarted","Data":"301d54b8aa1df9b232c4049abe9cc49ddce7fdd499af099eb58311d8d7409a0b"}
Oct 30 13:07:57.251286 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:57.251234    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt" podStartSLOduration=1.251218131 podStartE2EDuration="1.251218131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:57.250105557 +0000 UTC m=+1865.774389794" watchObservedRunningTime="2026-10-30 13:07:57.251218131 +0000 UTC m=+1865.775502350"
Oct 30 13:07:57.889097 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:57.889066    5837 scope.go:122] "RemoveContainer" containerID="a2e617ffed22f140849847408b9672fbff91594daa255e642b304fdd6b309369"
Oct 30 13:07:57.899096 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:57.899069    5837 scope.go:122] "RemoveContainer" containerID="bf518544cbbe5cb7f321b027352494e6e181a520960d6ccc7189f1c73a013b84"
Oct 30 13:07:57.908123 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:57.908093    5837 scope.go:122] "RemoveContainer" containerID="30bf9b7ccb6270b0097c12375ecab7528102b12fe443a704664bd837662198f2"
Oct 30 13:07:58.221045 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:58.220960    5837 generic.go:356] "Generic (PLEG): container finished" podID="6af65555-7415-4c48-beba-44e6ada79954" containerID="34079e242579d2d335185bf89daafa4a1a89dbbd88deab3364ea9133745abd42" exitCode=0
Oct 30 13:07:58.221045 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:58.221028    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt" event={"ID":"6af65555-7415-4c48-beba-44e6ada79954","Type":"ContainerDied","Data":"34079e242579d2d335185bf89daafa4a1a89dbbd88deab3364ea9133745abd42"}
Oct 30 13:07:59.391676 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:59.391650    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"
Oct 30 13:07:59.468483 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:59.468457    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af65555-7415-4c48-beba-44e6ada79954-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af65555-7415-4c48-beba-44e6ada79954-maas-api-openshift-ca\") pod \"6af65555-7415-4c48-beba-44e6ada79954\" (UID: \"6af65555-7415-4c48-beba-44e6ada79954\") "
Oct 30 13:07:59.468717 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:59.468576    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af65555-7415-4c48-beba-44e6ada79954-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af65555-7415-4c48-beba-44e6ada79954-maas-api-xks-ca\") pod \"6af65555-7415-4c48-beba-44e6ada79954\" (UID: \"6af65555-7415-4c48-beba-44e6ada79954\") "
Oct 30 13:07:59.468717 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:59.468680    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af65555-7415-4c48-beba-44e6ada79954-kube-api-access-ggt4c\" (UniqueName: \"kubernetes.io/projected/6af65555-7415-4c48-beba-44e6ada79954-kube-api-access-ggt4c\") pod \"6af65555-7415-4c48-beba-44e6ada79954\" (UID: \"6af65555-7415-4c48-beba-44e6ada79954\") "
Oct 30 13:07:59.468906 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:59.468869    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af65555-7415-4c48-beba-44e6ada79954-maas-api-openshift-ca" pod "6af65555-7415-4c48-beba-44e6ada79954" (UID: "6af65555-7415-4c48-beba-44e6ada79954"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:59.469223 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:59.469201    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af65555-7415-4c48-beba-44e6ada79954-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:59.470909 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:59.470881    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af65555-7415-4c48-beba-44e6ada79954-maas-api-xks-ca" pod "6af65555-7415-4c48-beba-44e6ada79954" (UID: "6af65555-7415-4c48-beba-44e6ada79954"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:07:59.471219 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:59.471198    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af65555-7415-4c48-beba-44e6ada79954-kube-api-access-ggt4c" pod "6af65555-7415-4c48-beba-44e6ada79954" (UID: "6af65555-7415-4c48-beba-44e6ada79954"). InnerVolumeSpecName "kube-api-access-ggt4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:59.570336 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:59.570307    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af65555-7415-4c48-beba-44e6ada79954-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:59.570336 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:59.570334    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggt4c\" (UniqueName: \"kubernetes.io/projected/6af65555-7415-4c48-beba-44e6ada79954-kube-api-access-ggt4c\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:07:59.959718 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:07:59.959686    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd_1d126169-6889-474a-a6e9-bda03dc1df4c/payload-processing/0.log"
Oct 30 13:08:00.235039 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:00.235009    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"
Oct 30 13:08:00.235236 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:00.235014    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt" event={"ID":"6af65555-7415-4c48-beba-44e6ada79954","Type":"ContainerDied","Data":"301d54b8aa1df9b232c4049abe9cc49ddce7fdd499af099eb58311d8d7409a0b"}
Oct 30 13:08:00.235236 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:00.235128    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="301d54b8aa1df9b232c4049abe9cc49ddce7fdd499af099eb58311d8d7409a0b"
Oct 30 13:08:00.554963 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:00.554894    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f_d630a8a0-5896-415e-8c51-34bcd9486545/payload-processing/0.log"
Oct 30 13:08:00.736144 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:00.736120    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f_d630a8a0-5896-415e-8c51-34bcd9486545/payload-processing/0.log"
Oct 30 13:08:01.326523 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.326492    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"]
Oct 30 13:08:01.327088 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.327074    5837 cpu_manager.go:414] "containerMap: removing container" podUID="6af65555-7415-4c48-beba-44e6ada79954" containerName="revoke-keys"
Oct 30 13:08:01.327145 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.327090    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af65555-7415-4c48-beba-44e6ada79954" containerName="revoke-keys"
Oct 30 13:08:01.327203 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.327195    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af65555-7415-4c48-beba-44e6ada79954" containerName="revoke-keys"
Oct 30 13:08:01.330839 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.330823    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"
Oct 30 13:08:01.346130 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.346107    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:08:01.357689 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.357666    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:08:01.357877 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.357833    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:08:01.357973 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.357863    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:08:01.362668 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.362644    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"]
Oct 30 13:08:01.399883 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.399858    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj\" (UID: \"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"
Oct 30 13:08:01.399996 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.399980    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq89r\" (UniqueName: \"kubernetes.io/projected/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-kube-api-access-qq89r\") pod \"maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj\" (UID: \"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"
Oct 30 13:08:01.400059 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.400022    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj\" (UID: \"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"
Oct 30 13:08:01.501597 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.501570    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj\" (UID: \"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"
Oct 30 13:08:01.501719 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.501633    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq89r\" (UniqueName: \"kubernetes.io/projected/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-kube-api-access-qq89r\") pod \"maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj\" (UID: \"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"
Oct 30 13:08:01.501719 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.501664    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj\" (UID: \"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"
Oct 30 13:08:01.502286 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.502267    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj\" (UID: \"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"
Oct 30 13:08:01.503984 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.503968    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj\" (UID: \"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"
Oct 30 13:08:01.522680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.522659    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq89r\" (UniqueName: \"kubernetes.io/projected/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-kube-api-access-qq89r\") pod \"maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj\" (UID: \"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"
Oct 30 13:08:01.648903 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.648829    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"
Oct 30 13:08:01.947638 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:01.947607    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"]
Oct 30 13:08:01.948905 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:08:01.948880    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd8aea7_8ccb_4ac5_855f_ddc3c5e9f258.slice/crio-a62c97a5c7e6f8176f86e77e9d1504e3f50f532e5b9f1f79f5f4d47525ef783b WatchSource:0}: Error finding container a62c97a5c7e6f8176f86e77e9d1504e3f50f532e5b9f1f79f5f4d47525ef783b: Status 404 returned error can't find the container with id a62c97a5c7e6f8176f86e77e9d1504e3f50f532e5b9f1f79f5f4d47525ef783b
Oct 30 13:08:02.250049 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:02.250014    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj" event={"ID":"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258","Type":"ContainerStarted","Data":"a62c97a5c7e6f8176f86e77e9d1504e3f50f532e5b9f1f79f5f4d47525ef783b"}
Oct 30 13:08:03.259279 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:03.259240    5837 generic.go:356] "Generic (PLEG): container finished" podID="0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258" containerID="8a5b8ef202970cba1f01232cca924d330765560935223bd534189d6994f61c80" exitCode=0
Oct 30 13:08:03.259755 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:03.259327    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj" event={"ID":"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258","Type":"ContainerDied","Data":"8a5b8ef202970cba1f01232cca924d330765560935223bd534189d6994f61c80"}
Oct 30 13:08:04.422321 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:04.422295    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"
Oct 30 13:08:04.538198 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:04.538122    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-kube-api-access-qq89r\" (UniqueName: \"kubernetes.io/projected/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-kube-api-access-qq89r\") pod \"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258\" (UID: \"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258\") "
Oct 30 13:08:04.538350 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:04.538240    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-maas-api-openshift-ca\") pod \"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258\" (UID: \"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258\") "
Oct 30 13:08:04.538350 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:04.538266    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-maas-api-xks-ca\") pod \"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258\" (UID: \"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258\") "
Oct 30 13:08:04.538594 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:04.538569    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-maas-api-openshift-ca" pod "0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258" (UID: "0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:04.538838 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:04.538822    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:04.540475 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:04.540449    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-maas-api-xks-ca" pod "0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258" (UID: "0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:04.540475 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:04.540471    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-kube-api-access-qq89r" pod "0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258" (UID: "0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258"). InnerVolumeSpecName "kube-api-access-qq89r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:04.640019 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:04.639995    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:04.640019 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:04.640021    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq89r\" (UniqueName: \"kubernetes.io/projected/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258-kube-api-access-qq89r\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:05.272962 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:05.272931    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"
Oct 30 13:08:05.272962 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:05.272944    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj" event={"ID":"0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258","Type":"ContainerDied","Data":"a62c97a5c7e6f8176f86e77e9d1504e3f50f532e5b9f1f79f5f4d47525ef783b"}
Oct 30 13:08:05.273180 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:05.272992    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a62c97a5c7e6f8176f86e77e9d1504e3f50f532e5b9f1f79f5f4d47525ef783b"
Oct 30 13:08:11.489883 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.489851    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"]
Oct 30 13:08:11.490515 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.490497    5837 cpu_manager.go:414] "containerMap: removing container" podUID="0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258" containerName="revoke-keys"
Oct 30 13:08:11.490515 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.490515    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258" containerName="revoke-keys"
Oct 30 13:08:11.490639 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.490629    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258" containerName="revoke-keys"
Oct 30 13:08:11.494267 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.494248    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"
Oct 30 13:08:11.511457 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.511433    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"]
Oct 30 13:08:11.520166 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.520144    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:08:11.520308 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.520153    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:08:11.520308 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.520165    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:08:11.520506 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.520171    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:08:11.643883 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.643852    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cb822c-3e4d-4db6-86e0-ca099e164daf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9699cd68-6rl6v\" (UID: \"f3cb822c-3e4d-4db6-86e0-ca099e164daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"
Oct 30 13:08:11.644075 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.643896    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82nml\" (UniqueName: \"kubernetes.io/projected/f3cb822c-3e4d-4db6-86e0-ca099e164daf-kube-api-access-82nml\") pod \"maas-api-revoke-sub-9699cd68-6rl6v\" (UID: \"f3cb822c-3e4d-4db6-86e0-ca099e164daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"
Oct 30 13:08:11.644167 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.644145    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cb822c-3e4d-4db6-86e0-ca099e164daf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9699cd68-6rl6v\" (UID: \"f3cb822c-3e4d-4db6-86e0-ca099e164daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"
Oct 30 13:08:11.745612 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.745543    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cb822c-3e4d-4db6-86e0-ca099e164daf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9699cd68-6rl6v\" (UID: \"f3cb822c-3e4d-4db6-86e0-ca099e164daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"
Oct 30 13:08:11.745612 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.745585    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82nml\" (UniqueName: \"kubernetes.io/projected/f3cb822c-3e4d-4db6-86e0-ca099e164daf-kube-api-access-82nml\") pod \"maas-api-revoke-sub-9699cd68-6rl6v\" (UID: \"f3cb822c-3e4d-4db6-86e0-ca099e164daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"
Oct 30 13:08:11.745821 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.745702    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cb822c-3e4d-4db6-86e0-ca099e164daf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9699cd68-6rl6v\" (UID: \"f3cb822c-3e4d-4db6-86e0-ca099e164daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"
Oct 30 13:08:11.746176 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.746156    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cb822c-3e4d-4db6-86e0-ca099e164daf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9699cd68-6rl6v\" (UID: \"f3cb822c-3e4d-4db6-86e0-ca099e164daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"
Oct 30 13:08:11.748056 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.748035    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cb822c-3e4d-4db6-86e0-ca099e164daf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9699cd68-6rl6v\" (UID: \"f3cb822c-3e4d-4db6-86e0-ca099e164daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"
Oct 30 13:08:11.765123 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.765100    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82nml\" (UniqueName: \"kubernetes.io/projected/f3cb822c-3e4d-4db6-86e0-ca099e164daf-kube-api-access-82nml\") pod \"maas-api-revoke-sub-9699cd68-6rl6v\" (UID: \"f3cb822c-3e4d-4db6-86e0-ca099e164daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"
Oct 30 13:08:11.807296 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:11.807272    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"
Oct 30 13:08:12.110685 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:12.110213    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"]
Oct 30 13:08:12.112449 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:08:12.112420    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cb822c_3e4d_4db6_86e0_ca099e164daf.slice/crio-21fd61ff631136df1bd3102ff2a8c4f7eb0ae6bad4c537e625376bbd62d6adfb WatchSource:0}: Error finding container 21fd61ff631136df1bd3102ff2a8c4f7eb0ae6bad4c537e625376bbd62d6adfb: Status 404 returned error can't find the container with id 21fd61ff631136df1bd3102ff2a8c4f7eb0ae6bad4c537e625376bbd62d6adfb
Oct 30 13:08:12.373605 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:12.373530    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v" event={"ID":"f3cb822c-3e4d-4db6-86e0-ca099e164daf","Type":"ContainerStarted","Data":"21fd61ff631136df1bd3102ff2a8c4f7eb0ae6bad4c537e625376bbd62d6adfb"}
Oct 30 13:08:13.390501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:13.390462    5837 generic.go:356] "Generic (PLEG): container finished" podID="f3cb822c-3e4d-4db6-86e0-ca099e164daf" containerID="3e65387fe6a97f296513a3030bc74704d5b9b30b9dfe251c5d6993e31a5086ea" exitCode=0
Oct 30 13:08:13.390993 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:13.390550    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v" event={"ID":"f3cb822c-3e4d-4db6-86e0-ca099e164daf","Type":"ContainerDied","Data":"3e65387fe6a97f296513a3030bc74704d5b9b30b9dfe251c5d6993e31a5086ea"}
Oct 30 13:08:14.553074 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:14.553049    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"
Oct 30 13:08:14.584442 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:14.584416    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cb822c-3e4d-4db6-86e0-ca099e164daf-kube-api-access-82nml\" (UniqueName: \"kubernetes.io/projected/f3cb822c-3e4d-4db6-86e0-ca099e164daf-kube-api-access-82nml\") pod \"f3cb822c-3e4d-4db6-86e0-ca099e164daf\" (UID: \"f3cb822c-3e4d-4db6-86e0-ca099e164daf\") "
Oct 30 13:08:14.584523 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:14.584502    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3cb822c-3e4d-4db6-86e0-ca099e164daf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cb822c-3e4d-4db6-86e0-ca099e164daf-maas-api-openshift-ca\") pod \"f3cb822c-3e4d-4db6-86e0-ca099e164daf\" (UID: \"f3cb822c-3e4d-4db6-86e0-ca099e164daf\") "
Oct 30 13:08:14.584559 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:14.584535    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3cb822c-3e4d-4db6-86e0-ca099e164daf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cb822c-3e4d-4db6-86e0-ca099e164daf-maas-api-xks-ca\") pod \"f3cb822c-3e4d-4db6-86e0-ca099e164daf\" (UID: \"f3cb822c-3e4d-4db6-86e0-ca099e164daf\") "
Oct 30 13:08:14.584822 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:14.584800    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cb822c-3e4d-4db6-86e0-ca099e164daf-maas-api-openshift-ca" pod "f3cb822c-3e4d-4db6-86e0-ca099e164daf" (UID: "f3cb822c-3e4d-4db6-86e0-ca099e164daf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:14.585067 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:14.585052    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cb822c-3e4d-4db6-86e0-ca099e164daf-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:14.586528 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:14.586507    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cb822c-3e4d-4db6-86e0-ca099e164daf-kube-api-access-82nml" pod "f3cb822c-3e4d-4db6-86e0-ca099e164daf" (UID: "f3cb822c-3e4d-4db6-86e0-ca099e164daf"). InnerVolumeSpecName "kube-api-access-82nml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:14.586702 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:14.586681    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3cb822c-3e4d-4db6-86e0-ca099e164daf-maas-api-xks-ca" pod "f3cb822c-3e4d-4db6-86e0-ca099e164daf" (UID: "f3cb822c-3e4d-4db6-86e0-ca099e164daf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:14.686230 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:14.686154    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82nml\" (UniqueName: \"kubernetes.io/projected/f3cb822c-3e4d-4db6-86e0-ca099e164daf-kube-api-access-82nml\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:14.686230 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:14.686180    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cb822c-3e4d-4db6-86e0-ca099e164daf-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:15.094868 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:08:15.094834    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:08:15.406087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:15.406008    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v" event={"ID":"f3cb822c-3e4d-4db6-86e0-ca099e164daf","Type":"ContainerDied","Data":"21fd61ff631136df1bd3102ff2a8c4f7eb0ae6bad4c537e625376bbd62d6adfb"}
Oct 30 13:08:15.406087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:15.406056    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21fd61ff631136df1bd3102ff2a8c4f7eb0ae6bad4c537e625376bbd62d6adfb"
Oct 30 13:08:15.406286 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:15.406109    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9699cd68-6rl6v"
Oct 30 13:08:21.173418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.173033    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"]
Oct 30 13:08:21.174053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.174015    5837 cpu_manager.go:414] "containerMap: removing container" podUID="f3cb822c-3e4d-4db6-86e0-ca099e164daf" containerName="revoke-keys"
Oct 30 13:08:21.174053 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.174038    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cb822c-3e4d-4db6-86e0-ca099e164daf" containerName="revoke-keys"
Oct 30 13:08:21.177416 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.174219    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cb822c-3e4d-4db6-86e0-ca099e164daf" containerName="revoke-keys"
Oct 30 13:08:21.226205 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.224854    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"
Oct 30 13:08:21.226554 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.226507    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"]
Oct 30 13:08:21.246488 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.246355    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:08:21.278915 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.266183    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:08:21.278915 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.266277    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:08:21.278915 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.266301    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:08:21.332716 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.305741    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-300b7a2e-hg845\" (UID: \"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"
Oct 30 13:08:21.332716 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.305882    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-300b7a2e-hg845\" (UID: \"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"
Oct 30 13:08:21.332716 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.305925    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4ch9\" (UniqueName: \"kubernetes.io/projected/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-kube-api-access-x4ch9\") pod \"maas-api-revoke-sub-300b7a2e-hg845\" (UID: \"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"
Oct 30 13:08:21.419439 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.407191    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-300b7a2e-hg845\" (UID: \"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"
Oct 30 13:08:21.419439 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.407360    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-300b7a2e-hg845\" (UID: \"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"
Oct 30 13:08:21.419439 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.407434    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4ch9\" (UniqueName: \"kubernetes.io/projected/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-kube-api-access-x4ch9\") pod \"maas-api-revoke-sub-300b7a2e-hg845\" (UID: \"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"
Oct 30 13:08:21.477468 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.474576    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-300b7a2e-hg845\" (UID: \"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"
Oct 30 13:08:21.483439 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.483366    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4ch9\" (UniqueName: \"kubernetes.io/projected/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-kube-api-access-x4ch9\") pod \"maas-api-revoke-sub-300b7a2e-hg845\" (UID: \"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"
Oct 30 13:08:21.519888 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.519856    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-300b7a2e-hg845\" (UID: \"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"
Oct 30 13:08:21.758421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:21.750663    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"
Oct 30 13:08:22.466521 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:22.436507    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:22.466521 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:22.440953    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:22.669509 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:22.600181    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:22.669509 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:22.631204    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"]
Oct 30 13:08:23.051666 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:23.051576    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845" event={"ID":"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5","Type":"ContainerStarted","Data":"c90e8d7721ce4d3a3719ca3df1008c0c323a6949dd0ca164e902d4570791b62b"}
Oct 30 13:08:23.209497 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:23.205374    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:23.449428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:23.444505    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/ready\": context deadline exceeded"
Oct 30 13:08:23.449428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:23.444929    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:08:23.449428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:23.445961    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": context deadline exceeded"
Oct 30 13:08:23.708946 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:23.631500    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:24.049794 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:24.049022    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:08:24.068854 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:24.068810    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845" event={"ID":"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5","Type":"ContainerStarted","Data":"038f1fdf9077b0e9bac7e97ddd715723b9999faeea92fd8504aedb33bd1c3004"}
Oct 30 13:08:25.075575 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:25.075539    5837 generic.go:356] "Generic (PLEG): container finished" podID="1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5" containerID="038f1fdf9077b0e9bac7e97ddd715723b9999faeea92fd8504aedb33bd1c3004" exitCode=0
Oct 30 13:08:25.075991 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:25.075628    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845" event={"ID":"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5","Type":"ContainerDied","Data":"038f1fdf9077b0e9bac7e97ddd715723b9999faeea92fd8504aedb33bd1c3004"}
Oct 30 13:08:26.226424 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:26.226367    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"
Oct 30 13:08:26.286427 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:26.286376    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-maas-api-openshift-ca\") pod \"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5\" (UID: \"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5\") "
Oct 30 13:08:26.286633 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:26.286585    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-kube-api-access-x4ch9\" (UniqueName: \"kubernetes.io/projected/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-kube-api-access-x4ch9\") pod \"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5\" (UID: \"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5\") "
Oct 30 13:08:26.286633 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:26.286618    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-maas-api-xks-ca\") pod \"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5\" (UID: \"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5\") "
Oct 30 13:08:26.286776 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:26.286733    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-maas-api-openshift-ca" pod "1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5" (UID: "1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:26.287096 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:26.287071    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:26.288950 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:26.288924    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-kube-api-access-x4ch9" pod "1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5" (UID: "1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5"). InnerVolumeSpecName "kube-api-access-x4ch9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:26.288950 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:26.288932    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-maas-api-xks-ca" pod "1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5" (UID: "1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:26.389068 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:26.388995    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4ch9\" (UniqueName: \"kubernetes.io/projected/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-kube-api-access-x4ch9\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:26.389068 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:26.389029    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:27.089821 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:27.089793    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845"
Oct 30 13:08:27.090018 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:27.089788    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300b7a2e-hg845" event={"ID":"1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5","Type":"ContainerDied","Data":"c90e8d7721ce4d3a3719ca3df1008c0c323a6949dd0ca164e902d4570791b62b"}
Oct 30 13:08:27.090018 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:27.089931    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c90e8d7721ce4d3a3719ca3df1008c0c323a6949dd0ca164e902d4570791b62b"
Oct 30 13:08:28.730113 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:28.730084    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"]
Oct 30 13:08:28.741896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:28.741869    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cb0fe-wt5gt"]
Oct 30 13:08:29.612707 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:29.612649    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af65555-7415-4c48-beba-44e6ada79954" path="/var/lib/kubelet/pods/6af65555-7415-4c48-beba-44e6ada79954/volumes"
Oct 30 13:08:34.208889 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:34.208803    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"]
Oct 30 13:08:36.563132 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:36.563099    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"]
Oct 30 13:08:36.563578 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:36.563332    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd" podUID="1d126169-6889-474a-a6e9-bda03dc1df4c" containerName="payload-processing" containerID="cri-o://ab50ec505bdca3e730fdc315557ca9d944c949aca8991e0668819d330a37eb27" gracePeriod=30
Oct 30 13:08:36.592316 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:36.592286    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"]
Oct 30 13:08:36.592559 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:36.592534    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56" podUID="5ae1fb09-7412-451d-b4d9-966a08372805" containerName="payload-pre-processing" containerID="cri-o://c864f3da0f5f426398aeb678cd9891e5150ea18505ae4904324a4db0236c207b" gracePeriod=30
Oct 30 13:08:36.918173 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:36.917853    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"
Oct 30 13:08:37.007928 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.007615    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"
Oct 30 13:08:37.024414 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.024363    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d126169-6889-474a-a6e9-bda03dc1df4c-kube-api-access-rst7c\" (UniqueName: \"kubernetes.io/projected/1d126169-6889-474a-a6e9-bda03dc1df4c-kube-api-access-rst7c\") pod \"1d126169-6889-474a-a6e9-bda03dc1df4c\" (UID: \"1d126169-6889-474a-a6e9-bda03dc1df4c\") "
Oct 30 13:08:37.024648 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.024628    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d126169-6889-474a-a6e9-bda03dc1df4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d126169-6889-474a-a6e9-bda03dc1df4c-plugins-config-volume\") pod \"1d126169-6889-474a-a6e9-bda03dc1df4c\" (UID: \"1d126169-6889-474a-a6e9-bda03dc1df4c\") "
Oct 30 13:08:37.025037 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.025008    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d126169-6889-474a-a6e9-bda03dc1df4c-plugins-config-volume" pod "1d126169-6889-474a-a6e9-bda03dc1df4c" (UID: "1d126169-6889-474a-a6e9-bda03dc1df4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:37.026847 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.026825    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d126169-6889-474a-a6e9-bda03dc1df4c-kube-api-access-rst7c" pod "1d126169-6889-474a-a6e9-bda03dc1df4c" (UID: "1d126169-6889-474a-a6e9-bda03dc1df4c"). InnerVolumeSpecName "kube-api-access-rst7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:37.126214 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.126142    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae1fb09-7412-451d-b4d9-966a08372805-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae1fb09-7412-451d-b4d9-966a08372805-plugins-config-volume\") pod \"5ae1fb09-7412-451d-b4d9-966a08372805\" (UID: \"5ae1fb09-7412-451d-b4d9-966a08372805\") "
Oct 30 13:08:37.126214 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.126184    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae1fb09-7412-451d-b4d9-966a08372805-kube-api-access-p8xls\" (UniqueName: \"kubernetes.io/projected/5ae1fb09-7412-451d-b4d9-966a08372805-kube-api-access-p8xls\") pod \"5ae1fb09-7412-451d-b4d9-966a08372805\" (UID: \"5ae1fb09-7412-451d-b4d9-966a08372805\") "
Oct 30 13:08:37.126526 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.126505    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae1fb09-7412-451d-b4d9-966a08372805-plugins-config-volume" pod "5ae1fb09-7412-451d-b4d9-966a08372805" (UID: "5ae1fb09-7412-451d-b4d9-966a08372805"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:37.126661 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.126647    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d126169-6889-474a-a6e9-bda03dc1df4c-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:37.126702 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.126665    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rst7c\" (UniqueName: \"kubernetes.io/projected/1d126169-6889-474a-a6e9-bda03dc1df4c-kube-api-access-rst7c\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:37.126702 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.126674    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae1fb09-7412-451d-b4d9-966a08372805-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:37.128530 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.128508    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae1fb09-7412-451d-b4d9-966a08372805-kube-api-access-p8xls" pod "5ae1fb09-7412-451d-b4d9-966a08372805" (UID: "5ae1fb09-7412-451d-b4d9-966a08372805"). InnerVolumeSpecName "kube-api-access-p8xls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:37.161452 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.161381    5837 generic.go:356] "Generic (PLEG): container finished" podID="1d126169-6889-474a-a6e9-bda03dc1df4c" containerID="ab50ec505bdca3e730fdc315557ca9d944c949aca8991e0668819d330a37eb27" exitCode=0
Oct 30 13:08:37.161452 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.161444    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"
Oct 30 13:08:37.161633 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.161470    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd" event={"ID":"1d126169-6889-474a-a6e9-bda03dc1df4c","Type":"ContainerDied","Data":"ab50ec505bdca3e730fdc315557ca9d944c949aca8991e0668819d330a37eb27"}
Oct 30 13:08:37.161633 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.161536    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd" event={"ID":"1d126169-6889-474a-a6e9-bda03dc1df4c","Type":"ContainerDied","Data":"f0596c67a1a2113efdce5e194af77369fe495713da14f71810113d00f049dfbd"}
Oct 30 13:08:37.161633 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.161566    5837 scope.go:122] "RemoveContainer" containerID="ab50ec505bdca3e730fdc315557ca9d944c949aca8991e0668819d330a37eb27"
Oct 30 13:08:37.163271 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.163247    5837 generic.go:356] "Generic (PLEG): container finished" podID="5ae1fb09-7412-451d-b4d9-966a08372805" containerID="c864f3da0f5f426398aeb678cd9891e5150ea18505ae4904324a4db0236c207b" exitCode=0
Oct 30 13:08:37.163353 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.163295    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"
Oct 30 13:08:37.163353 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.163303    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56" event={"ID":"5ae1fb09-7412-451d-b4d9-966a08372805","Type":"ContainerDied","Data":"c864f3da0f5f426398aeb678cd9891e5150ea18505ae4904324a4db0236c207b"}
Oct 30 13:08:37.163353 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.163345    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56" event={"ID":"5ae1fb09-7412-451d-b4d9-966a08372805","Type":"ContainerDied","Data":"020286d0f7650ba2152f041141523ac0294c951aaa0a282b675cc7a93718d56f"}
Oct 30 13:08:37.172866 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.172845    5837 scope.go:122] "RemoveContainer" containerID="ab50ec505bdca3e730fdc315557ca9d944c949aca8991e0668819d330a37eb27"
Oct 30 13:08:37.173238 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:08:37.173161    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab50ec505bdca3e730fdc315557ca9d944c949aca8991e0668819d330a37eb27\": container with ID starting with ab50ec505bdca3e730fdc315557ca9d944c949aca8991e0668819d330a37eb27 not found: ID does not exist" containerID="ab50ec505bdca3e730fdc315557ca9d944c949aca8991e0668819d330a37eb27"
Oct 30 13:08:37.173238 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.173187    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab50ec505bdca3e730fdc315557ca9d944c949aca8991e0668819d330a37eb27"} err="failed to get container status \"ab50ec505bdca3e730fdc315557ca9d944c949aca8991e0668819d330a37eb27\": rpc error: code = NotFound desc = could not find container \"ab50ec505bdca3e730fdc315557ca9d944c949aca8991e0668819d330a37eb27\": container with ID starting with ab50ec505bdca3e730fdc315557ca9d944c949aca8991e0668819d330a37eb27 not found: ID does not exist"
Oct 30 13:08:37.173238 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.173209    5837 scope.go:122] "RemoveContainer" containerID="c864f3da0f5f426398aeb678cd9891e5150ea18505ae4904324a4db0236c207b"
Oct 30 13:08:37.182761 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.182736    5837 scope.go:122] "RemoveContainer" containerID="c864f3da0f5f426398aeb678cd9891e5150ea18505ae4904324a4db0236c207b"
Oct 30 13:08:37.183015 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:08:37.183000    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c864f3da0f5f426398aeb678cd9891e5150ea18505ae4904324a4db0236c207b\": container with ID starting with c864f3da0f5f426398aeb678cd9891e5150ea18505ae4904324a4db0236c207b not found: ID does not exist" containerID="c864f3da0f5f426398aeb678cd9891e5150ea18505ae4904324a4db0236c207b"
Oct 30 13:08:37.183077 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.183020    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c864f3da0f5f426398aeb678cd9891e5150ea18505ae4904324a4db0236c207b"} err="failed to get container status \"c864f3da0f5f426398aeb678cd9891e5150ea18505ae4904324a4db0236c207b\": rpc error: code = NotFound desc = could not find container \"c864f3da0f5f426398aeb678cd9891e5150ea18505ae4904324a4db0236c207b\": container with ID starting with c864f3da0f5f426398aeb678cd9891e5150ea18505ae4904324a4db0236c207b not found: ID does not exist"
Oct 30 13:08:37.203678 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.203655    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"]
Oct 30 13:08:37.206681 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.206661    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-026c2f-wc9pj"]
Oct 30 13:08:37.215135 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.215115    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"]
Oct 30 13:08:37.219855 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.219756    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-026c2f-5f47c6c65b-sm9gd"]
Oct 30 13:08:37.227839 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.227821    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8xls\" (UniqueName: \"kubernetes.io/projected/5ae1fb09-7412-451d-b4d9-966a08372805-kube-api-access-p8xls\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:37.257516 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.257486    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"]
Oct 30 13:08:37.260119 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.260098    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c68c9868c-p8k56"]
Oct 30 13:08:37.606189 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.606156    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258" path="/var/lib/kubelet/pods/0fd8aea7-8ccb-4ac5-855f-ddc3c5e9f258/volumes"
Oct 30 13:08:37.606581 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.606566    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d126169-6889-474a-a6e9-bda03dc1df4c" path="/var/lib/kubelet/pods/1d126169-6889-474a-a6e9-bda03dc1df4c/volumes"
Oct 30 13:08:37.606914 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:37.606900    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae1fb09-7412-451d-b4d9-966a08372805" path="/var/lib/kubelet/pods/5ae1fb09-7412-451d-b4d9-966a08372805/volumes"
Oct 30 13:08:42.643675 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:42.643642    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"]
Oct 30 13:08:45.093500 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:08:45.093465    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:08:49.014422 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:49.011480    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:08:49.298479 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:49.285284    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main" containerID="cri-o://e42d9cad13590bb8d372bffac3c60c7f6827a1d1908cee1561dbc672c0291449" gracePeriod=45
Oct 30 13:08:49.443541 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:49.443486    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:49.444477 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:49.444450    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": dial tcp 10.134.0.200:8000: connect: connection refused"
Oct 30 13:08:49.444697 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:49.444682    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:08:49.538534 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:49.538485    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:49.622603 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:49.622496    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:50.021757 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:50.021711    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:50.024793 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:50.022912    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:08:50.301568 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:50.290030    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:08:50.443556 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:50.443501    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": dial tcp 10.134.0.200:8000: connect: connection refused"
Oct 30 13:08:50.547995 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:50.547874    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:50.966677 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:50.966656    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:08:51.109876 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.109845    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-tmp-dir\") pod \"bb128524-fe08-4334-8ffe-86220d1e62b8\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") "
Oct 30 13:08:51.110299 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.109995    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-home\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-home\") pod \"bb128524-fe08-4334-8ffe-86220d1e62b8\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") "
Oct 30 13:08:51.110299 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.110029    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-dshm\") pod \"bb128524-fe08-4334-8ffe-86220d1e62b8\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") "
Oct 30 13:08:51.110299 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.110062    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb128524-fe08-4334-8ffe-86220d1e62b8-kube-api-access-jlzr7\" (UniqueName: \"kubernetes.io/projected/bb128524-fe08-4334-8ffe-86220d1e62b8-kube-api-access-jlzr7\") pod \"bb128524-fe08-4334-8ffe-86220d1e62b8\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") "
Oct 30 13:08:51.110299 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.110080    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-tmp-dir" pod "bb128524-fe08-4334-8ffe-86220d1e62b8" (UID: "bb128524-fe08-4334-8ffe-86220d1e62b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:51.110299 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.110132    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb128524-fe08-4334-8ffe-86220d1e62b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb128524-fe08-4334-8ffe-86220d1e62b8-tls-certs\") pod \"bb128524-fe08-4334-8ffe-86220d1e62b8\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") "
Oct 30 13:08:51.110299 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.110167    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-model-cache\") pod \"bb128524-fe08-4334-8ffe-86220d1e62b8\" (UID: \"bb128524-fe08-4334-8ffe-86220d1e62b8\") "
Oct 30 13:08:51.110299 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.110259    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-home" pod "bb128524-fe08-4334-8ffe-86220d1e62b8" (UID: "bb128524-fe08-4334-8ffe-86220d1e62b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:51.110677 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.110423    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-model-cache" pod "bb128524-fe08-4334-8ffe-86220d1e62b8" (UID: "bb128524-fe08-4334-8ffe-86220d1e62b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:51.110677 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.110656    5837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-home\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:51.110677 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.110675    5837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-model-cache\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:51.110782 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.110686    5837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-tmp-dir\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:51.113279 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.113248    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-dshm" pod "bb128524-fe08-4334-8ffe-86220d1e62b8" (UID: "bb128524-fe08-4334-8ffe-86220d1e62b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:51.117077 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.116941    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb128524-fe08-4334-8ffe-86220d1e62b8-kube-api-access-jlzr7" pod "bb128524-fe08-4334-8ffe-86220d1e62b8" (UID: "bb128524-fe08-4334-8ffe-86220d1e62b8"). InnerVolumeSpecName "kube-api-access-jlzr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:51.117077 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.116977    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb128524-fe08-4334-8ffe-86220d1e62b8-tls-certs" pod "bb128524-fe08-4334-8ffe-86220d1e62b8" (UID: "bb128524-fe08-4334-8ffe-86220d1e62b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:51.211877 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.211823    5837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb128524-fe08-4334-8ffe-86220d1e62b8-dshm\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:51.211877 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.211846    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlzr7\" (UniqueName: \"kubernetes.io/projected/bb128524-fe08-4334-8ffe-86220d1e62b8-kube-api-access-jlzr7\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:51.211877 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.211857    5837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb128524-fe08-4334-8ffe-86220d1e62b8-tls-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:51.328480 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.328443    5837 generic.go:356] "Generic (PLEG): container finished" podID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerID="e42d9cad13590bb8d372bffac3c60c7f6827a1d1908cee1561dbc672c0291449" exitCode=0
Oct 30 13:08:51.328628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.328483    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"
Oct 30 13:08:51.328628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.328543    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" event={"ID":"bb128524-fe08-4334-8ffe-86220d1e62b8","Type":"ContainerDied","Data":"e42d9cad13590bb8d372bffac3c60c7f6827a1d1908cee1561dbc672c0291449"}
Oct 30 13:08:51.328628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.328581    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54" event={"ID":"bb128524-fe08-4334-8ffe-86220d1e62b8","Type":"ContainerDied","Data":"babcdf498564b59e679e90cdcf0db44e651e1f9eae5f76f2c039ac1122f44ea6"}
Oct 30 13:08:51.328628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.328597    5837 scope.go:122] "RemoveContainer" containerID="e42d9cad13590bb8d372bffac3c60c7f6827a1d1908cee1561dbc672c0291449"
Oct 30 13:08:51.339002 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.338986    5837 scope.go:122] "RemoveContainer" containerID="e42d9cad13590bb8d372bffac3c60c7f6827a1d1908cee1561dbc672c0291449"
Oct 30 13:08:51.339252 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:08:51.339236    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42d9cad13590bb8d372bffac3c60c7f6827a1d1908cee1561dbc672c0291449\": container with ID starting with e42d9cad13590bb8d372bffac3c60c7f6827a1d1908cee1561dbc672c0291449 not found: ID does not exist" containerID="e42d9cad13590bb8d372bffac3c60c7f6827a1d1908cee1561dbc672c0291449"
Oct 30 13:08:51.339315 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.339261    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42d9cad13590bb8d372bffac3c60c7f6827a1d1908cee1561dbc672c0291449"} err="failed to get container status \"e42d9cad13590bb8d372bffac3c60c7f6827a1d1908cee1561dbc672c0291449\": rpc error: code = NotFound desc = could not find container \"e42d9cad13590bb8d372bffac3c60c7f6827a1d1908cee1561dbc672c0291449\": container with ID starting with e42d9cad13590bb8d372bffac3c60c7f6827a1d1908cee1561dbc672c0291449 not found: ID does not exist"
Oct 30 13:08:51.383989 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.383956    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"]
Oct 30 13:08:51.395916 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.395894    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-4cb0fe/praxis-route-4cb0fe-kserve-754974d7b5-zbp54"]
Oct 30 13:08:51.605116 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:51.605086    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" path="/var/lib/kubelet/pods/bb128524-fe08-4334-8ffe-86220d1e62b8/volumes"
Oct 30 13:08:57.664428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:57.664363    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main" containerID="cri-o://555cff26e076f709f3a0104378a2d1113a493acd129059308eabb3990087764b" gracePeriod=45
Oct 30 13:08:57.927217 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:57.927151    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:08:57.981758 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:57.981729    5837 scope.go:122] "RemoveContainer" containerID="58ff2d91238c0e6e49ede3337e42165fdbd43e4f07fade7782f866636fb1431d"
Oct 30 13:08:57.991846 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:57.991820    5837 scope.go:122] "RemoveContainer" containerID="61154274d24043eccac022f5e2bd5401e2847e6a1c5b72cdb845e740d39479ba"
Oct 30 13:08:58.002075 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.002048    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-dshm\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-dshm\") pod \"403d9d4d-2f47-4d38-96de-32158a75a865\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") "
Oct 30 13:08:58.002228 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.002149    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-model-cache\") pod \"403d9d4d-2f47-4d38-96de-32158a75a865\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") "
Oct 30 13:08:58.002228 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.002212    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-tmp-dir\") pod \"403d9d4d-2f47-4d38-96de-32158a75a865\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") "
Oct 30 13:08:58.002362 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.002259    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/403d9d4d-2f47-4d38-96de-32158a75a865-tls-certs\" (UniqueName: \"kubernetes.io/secret/403d9d4d-2f47-4d38-96de-32158a75a865-tls-certs\") pod \"403d9d4d-2f47-4d38-96de-32158a75a865\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") "
Oct 30 13:08:58.002362 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.002345    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-home\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-home\") pod \"403d9d4d-2f47-4d38-96de-32158a75a865\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") "
Oct 30 13:08:58.002518 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.002462    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-model-cache" pod "403d9d4d-2f47-4d38-96de-32158a75a865" (UID: "403d9d4d-2f47-4d38-96de-32158a75a865"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:58.002518 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.002476    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403d9d4d-2f47-4d38-96de-32158a75a865-kube-api-access-8tsqw\" (UniqueName: \"kubernetes.io/projected/403d9d4d-2f47-4d38-96de-32158a75a865-kube-api-access-8tsqw\") pod \"403d9d4d-2f47-4d38-96de-32158a75a865\" (UID: \"403d9d4d-2f47-4d38-96de-32158a75a865\") "
Oct 30 13:08:58.002738 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.002716    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-home" pod "403d9d4d-2f47-4d38-96de-32158a75a865" (UID: "403d9d4d-2f47-4d38-96de-32158a75a865"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:58.002973 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.002901    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-tmp-dir" pod "403d9d4d-2f47-4d38-96de-32158a75a865" (UID: "403d9d4d-2f47-4d38-96de-32158a75a865"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:58.003184 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.003166    5837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-home\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:58.005149 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.005127    5837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-model-cache\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:58.005299 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.005285    5837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-tmp-dir\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:58.006816 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.006792    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403d9d4d-2f47-4d38-96de-32158a75a865-tls-certs" pod "403d9d4d-2f47-4d38-96de-32158a75a865" (UID: "403d9d4d-2f47-4d38-96de-32158a75a865"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:58.006935 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.006828    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-dshm" pod "403d9d4d-2f47-4d38-96de-32158a75a865" (UID: "403d9d4d-2f47-4d38-96de-32158a75a865"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:58.007004 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.006941    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403d9d4d-2f47-4d38-96de-32158a75a865-kube-api-access-8tsqw" pod "403d9d4d-2f47-4d38-96de-32158a75a865" (UID: "403d9d4d-2f47-4d38-96de-32158a75a865"). InnerVolumeSpecName "kube-api-access-8tsqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:58.106473 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.106439    5837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/403d9d4d-2f47-4d38-96de-32158a75a865-tls-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:58.106473 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.106470    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tsqw\" (UniqueName: \"kubernetes.io/projected/403d9d4d-2f47-4d38-96de-32158a75a865-kube-api-access-8tsqw\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:58.106674 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.106482    5837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/403d9d4d-2f47-4d38-96de-32158a75a865-dshm\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:08:58.408489 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.408452    5837 generic.go:356] "Generic (PLEG): container finished" podID="403d9d4d-2f47-4d38-96de-32158a75a865" containerID="555cff26e076f709f3a0104378a2d1113a493acd129059308eabb3990087764b" exitCode=0
Oct 30 13:08:58.408667 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.408514    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"
Oct 30 13:08:58.408667 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.408514    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" event={"ID":"403d9d4d-2f47-4d38-96de-32158a75a865","Type":"ContainerDied","Data":"555cff26e076f709f3a0104378a2d1113a493acd129059308eabb3990087764b"}
Oct 30 13:08:58.408667 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.408630    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd" event={"ID":"403d9d4d-2f47-4d38-96de-32158a75a865","Type":"ContainerDied","Data":"dcc6e371922349336679aa879b1fd5b110d472b5983826246c1079c6fc07e39d"}
Oct 30 13:08:58.408667 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.408649    5837 scope.go:122] "RemoveContainer" containerID="555cff26e076f709f3a0104378a2d1113a493acd129059308eabb3990087764b"
Oct 30 13:08:58.418721 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.418703    5837 scope.go:122] "RemoveContainer" containerID="555cff26e076f709f3a0104378a2d1113a493acd129059308eabb3990087764b"
Oct 30 13:08:58.418978 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:08:58.418960    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"555cff26e076f709f3a0104378a2d1113a493acd129059308eabb3990087764b\": container with ID starting with 555cff26e076f709f3a0104378a2d1113a493acd129059308eabb3990087764b not found: ID does not exist" containerID="555cff26e076f709f3a0104378a2d1113a493acd129059308eabb3990087764b"
Oct 30 13:08:58.419040 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.418985    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"555cff26e076f709f3a0104378a2d1113a493acd129059308eabb3990087764b"} err="failed to get container status \"555cff26e076f709f3a0104378a2d1113a493acd129059308eabb3990087764b\": rpc error: code = NotFound desc = could not find container \"555cff26e076f709f3a0104378a2d1113a493acd129059308eabb3990087764b\": container with ID starting with 555cff26e076f709f3a0104378a2d1113a493acd129059308eabb3990087764b not found: ID does not exist"
Oct 30 13:08:58.458249 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.458225    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"]
Oct 30 13:08:58.464267 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:58.464242    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-026c2f/ipp-route-026c2f-kserve-b495b7649-ljhhd"]
Oct 30 13:08:59.604253 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:08:59.604222    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" path="/var/lib/kubelet/pods/403d9d4d-2f47-4d38-96de-32158a75a865/volumes"
Oct 30 13:09:15.122880 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:09:15.122841    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:09:18.362089 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.362057    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"]
Oct 30 13:09:18.362521 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.362303    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b" podUID="4ff36816-2f3d-4007-b635-2a79923951f1" containerName="istio-proxy" containerID="cri-o://be0bf6100b94a83d02cc48323450ac52747455527d0d141b714a8144aa0e3211" gracePeriod=30
Oct 30 13:09:18.957425 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.957357    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"]
Oct 30 13:09:18.957954 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.957931    5837 cpu_manager.go:414] "containerMap: removing container" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main"
Oct 30 13:09:18.957954 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.957948    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main"
Oct 30 13:09:18.958113 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.957972    5837 cpu_manager.go:414] "containerMap: removing container" podUID="5ae1fb09-7412-451d-b4d9-966a08372805" containerName="payload-pre-processing"
Oct 30 13:09:18.958113 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.957981    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae1fb09-7412-451d-b4d9-966a08372805" containerName="payload-pre-processing"
Oct 30 13:09:18.958113 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.957998    5837 cpu_manager.go:414] "containerMap: removing container" podUID="1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5" containerName="revoke-keys"
Oct 30 13:09:18.958113 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.958003    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5" containerName="revoke-keys"
Oct 30 13:09:18.958113 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.958015    5837 cpu_manager.go:414] "containerMap: removing container" podUID="1d126169-6889-474a-a6e9-bda03dc1df4c" containerName="payload-processing"
Oct 30 13:09:18.958113 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.958020    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d126169-6889-474a-a6e9-bda03dc1df4c" containerName="payload-processing"
Oct 30 13:09:18.958113 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.958032    5837 cpu_manager.go:414] "containerMap: removing container" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main"
Oct 30 13:09:18.958113 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.958037    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main"
Oct 30 13:09:18.958358 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.958122    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="403d9d4d-2f47-4d38-96de-32158a75a865" containerName="main"
Oct 30 13:09:18.958358 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.958138    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb128524-fe08-4334-8ffe-86220d1e62b8" containerName="main"
Oct 30 13:09:18.958358 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.958149    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae1fb09-7412-451d-b4d9-966a08372805" containerName="payload-pre-processing"
Oct 30 13:09:18.958358 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.958155    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5" containerName="revoke-keys"
Oct 30 13:09:18.958358 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.958164    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d126169-6889-474a-a6e9-bda03dc1df4c" containerName="payload-processing"
Oct 30 13:09:18.961722 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.961707    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"
Oct 30 13:09:18.977788 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.977765    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:09:18.977788 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.977781    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:09:18.980851 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:18.980827    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"]
Oct 30 13:09:19.032304 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:19.032273    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:09:19.032557 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:19.032541    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:09:19.109613 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:19.109583    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6247fe5-c255-44b4-9186-c68883bef67b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5\" (UID: \"d6247fe5-c255-44b4-9186-c68883bef67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"
Oct 30 13:09:19.109613 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:19.109620    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqt85\" (UniqueName: \"kubernetes.io/projected/d6247fe5-c255-44b4-9186-c68883bef67b-kube-api-access-mqt85\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5\" (UID: \"d6247fe5-c255-44b4-9186-c68883bef67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"
Oct 30 13:09:19.109845 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:19.109779    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6247fe5-c255-44b4-9186-c68883bef67b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5\" (UID: \"d6247fe5-c255-44b4-9186-c68883bef67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"
Oct 30 13:09:19.211436 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:19.211337    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6247fe5-c255-44b4-9186-c68883bef67b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5\" (UID: \"d6247fe5-c255-44b4-9186-c68883bef67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"
Oct 30 13:09:19.211436 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:19.211370    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqt85\" (UniqueName: \"kubernetes.io/projected/d6247fe5-c255-44b4-9186-c68883bef67b-kube-api-access-mqt85\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5\" (UID: \"d6247fe5-c255-44b4-9186-c68883bef67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"
Oct 30 13:09:19.211647 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:19.211470    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6247fe5-c255-44b4-9186-c68883bef67b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5\" (UID: \"d6247fe5-c255-44b4-9186-c68883bef67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"
Oct 30 13:09:19.212065 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:19.212048    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6247fe5-c255-44b4-9186-c68883bef67b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5\" (UID: \"d6247fe5-c255-44b4-9186-c68883bef67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"
Oct 30 13:09:19.213779 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:19.213763    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6247fe5-c255-44b4-9186-c68883bef67b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5\" (UID: \"d6247fe5-c255-44b4-9186-c68883bef67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"
Oct 30 13:09:19.231664 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:19.231641    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqt85\" (UniqueName: \"kubernetes.io/projected/d6247fe5-c255-44b4-9186-c68883bef67b-kube-api-access-mqt85\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5\" (UID: \"d6247fe5-c255-44b4-9186-c68883bef67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"
Oct 30 13:09:19.275578 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:19.275552    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"
Oct 30 13:09:19.576355 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:19.576320    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"]
Oct 30 13:09:19.579044 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:09:19.579015    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6247fe5_c255_44b4_9186_c68883bef67b.slice/crio-69c0838626a562cf75a1db275e0eaf9455ad4dfc351483a3987e026f0a5a4f66 WatchSource:0}: Error finding container 69c0838626a562cf75a1db275e0eaf9455ad4dfc351483a3987e026f0a5a4f66: Status 404 returned error can't find the container with id 69c0838626a562cf75a1db275e0eaf9455ad4dfc351483a3987e026f0a5a4f66
Oct 30 13:09:20.588705 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:20.588663    5837 generic.go:356] "Generic (PLEG): container finished" podID="d6247fe5-c255-44b4-9186-c68883bef67b" containerID="5609149869af2d8675acf243641c5800b5123ffb39b5353427979277b52b3bd4" exitCode=0
Oct 30 13:09:20.589071 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:20.588749    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5" event={"ID":"d6247fe5-c255-44b4-9186-c68883bef67b","Type":"ContainerDied","Data":"5609149869af2d8675acf243641c5800b5123ffb39b5353427979277b52b3bd4"}
Oct 30 13:09:20.589071 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:20.588797    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5" event={"ID":"d6247fe5-c255-44b4-9186-c68883bef67b","Type":"ContainerStarted","Data":"69c0838626a562cf75a1db275e0eaf9455ad4dfc351483a3987e026f0a5a4f66"}
Oct 30 13:09:21.748845 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:21.748825    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"
Oct 30 13:09:21.845677 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:21.845647    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6247fe5-c255-44b4-9186-c68883bef67b-kube-api-access-mqt85\" (UniqueName: \"kubernetes.io/projected/d6247fe5-c255-44b4-9186-c68883bef67b-kube-api-access-mqt85\") pod \"d6247fe5-c255-44b4-9186-c68883bef67b\" (UID: \"d6247fe5-c255-44b4-9186-c68883bef67b\") "
Oct 30 13:09:21.845842 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:21.845705    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6247fe5-c255-44b4-9186-c68883bef67b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6247fe5-c255-44b4-9186-c68883bef67b-maas-api-openshift-ca\") pod \"d6247fe5-c255-44b4-9186-c68883bef67b\" (UID: \"d6247fe5-c255-44b4-9186-c68883bef67b\") "
Oct 30 13:09:21.845931 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:21.845889    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6247fe5-c255-44b4-9186-c68883bef67b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6247fe5-c255-44b4-9186-c68883bef67b-maas-api-xks-ca\") pod \"d6247fe5-c255-44b4-9186-c68883bef67b\" (UID: \"d6247fe5-c255-44b4-9186-c68883bef67b\") "
Oct 30 13:09:21.846117 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:21.846093    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6247fe5-c255-44b4-9186-c68883bef67b-maas-api-openshift-ca" pod "d6247fe5-c255-44b4-9186-c68883bef67b" (UID: "d6247fe5-c255-44b4-9186-c68883bef67b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:21.846238 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:21.846222    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6247fe5-c255-44b4-9186-c68883bef67b-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:21.847873 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:21.847816    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6247fe5-c255-44b4-9186-c68883bef67b-kube-api-access-mqt85" pod "d6247fe5-c255-44b4-9186-c68883bef67b" (UID: "d6247fe5-c255-44b4-9186-c68883bef67b"). InnerVolumeSpecName "kube-api-access-mqt85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:21.847962 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:21.847929    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6247fe5-c255-44b4-9186-c68883bef67b-maas-api-xks-ca" pod "d6247fe5-c255-44b4-9186-c68883bef67b" (UID: "d6247fe5-c255-44b4-9186-c68883bef67b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:21.946954 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:21.946917    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6247fe5-c255-44b4-9186-c68883bef67b-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:21.946954 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:21.946948    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqt85\" (UniqueName: \"kubernetes.io/projected/d6247fe5-c255-44b4-9186-c68883bef67b-kube-api-access-mqt85\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:22.604056 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:22.604027    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"
Oct 30 13:09:22.604239 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:22.604064    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5" event={"ID":"d6247fe5-c255-44b4-9186-c68883bef67b","Type":"ContainerDied","Data":"69c0838626a562cf75a1db275e0eaf9455ad4dfc351483a3987e026f0a5a4f66"}
Oct 30 13:09:22.604239 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:22.604113    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69c0838626a562cf75a1db275e0eaf9455ad4dfc351483a3987e026f0a5a4f66"
Oct 30 13:09:23.611575 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.611544    5837 generic.go:356] "Generic (PLEG): container finished" podID="4ff36816-2f3d-4007-b635-2a79923951f1" containerID="be0bf6100b94a83d02cc48323450ac52747455527d0d141b714a8144aa0e3211" exitCode=0
Oct 30 13:09:23.611997 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.611631    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b" event={"ID":"4ff36816-2f3d-4007-b635-2a79923951f1","Type":"ContainerDied","Data":"be0bf6100b94a83d02cc48323450ac52747455527d0d141b714a8144aa0e3211"}
Oct 30 13:09:23.611997 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.611672    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b" event={"ID":"4ff36816-2f3d-4007-b635-2a79923951f1","Type":"ContainerDied","Data":"fd2ef9555f65c7678708077b779b99e73e0d6f509c766ff558d5b7bda69bcb83"}
Oct 30 13:09:23.611997 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.611694    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd2ef9555f65c7678708077b779b99e73e0d6f509c766ff558d5b7bda69bcb83"
Oct 30 13:09:23.625656 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.625632    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:09:23.773052 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.773025    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-credential-socket\") pod \"4ff36816-2f3d-4007-b635-2a79923951f1\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") "
Oct 30 13:09:23.773248 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.773091    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ff36816-2f3d-4007-b635-2a79923951f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ff36816-2f3d-4007-b635-2a79923951f1-istio-podinfo\") pod \"4ff36816-2f3d-4007-b635-2a79923951f1\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") "
Oct 30 13:09:23.773248 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.773131    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ff36816-2f3d-4007-b635-2a79923951f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ff36816-2f3d-4007-b635-2a79923951f1-istiod-ca-cert\") pod \"4ff36816-2f3d-4007-b635-2a79923951f1\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") "
Oct 30 13:09:23.773248 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.773159    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-istio-envoy\") pod \"4ff36816-2f3d-4007-b635-2a79923951f1\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") "
Oct 30 13:09:23.773248 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.773218    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-istio-data\") pod \"4ff36816-2f3d-4007-b635-2a79923951f1\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") "
Oct 30 13:09:23.773248 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.773243    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ff36816-2f3d-4007-b635-2a79923951f1-kube-api-access-hhcr9\" (UniqueName: \"kubernetes.io/projected/4ff36816-2f3d-4007-b635-2a79923951f1-kube-api-access-hhcr9\") pod \"4ff36816-2f3d-4007-b635-2a79923951f1\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") "
Oct 30 13:09:23.773575 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.773264    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ff36816-2f3d-4007-b635-2a79923951f1-istio-token\" (UniqueName: \"kubernetes.io/projected/4ff36816-2f3d-4007-b635-2a79923951f1-istio-token\") pod \"4ff36816-2f3d-4007-b635-2a79923951f1\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") "
Oct 30 13:09:23.773575 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.773280    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-credential-socket" pod "4ff36816-2f3d-4007-b635-2a79923951f1" (UID: "4ff36816-2f3d-4007-b635-2a79923951f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:23.773575 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.773373    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-workload-socket\") pod \"4ff36816-2f3d-4007-b635-2a79923951f1\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") "
Oct 30 13:09:23.773575 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.773457    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-workload-certs\") pod \"4ff36816-2f3d-4007-b635-2a79923951f1\" (UID: \"4ff36816-2f3d-4007-b635-2a79923951f1\") "
Oct 30 13:09:23.773575 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.773542    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-istio-data" pod "4ff36816-2f3d-4007-b635-2a79923951f1" (UID: "4ff36816-2f3d-4007-b635-2a79923951f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:23.773864 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.773715    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-workload-socket" pod "4ff36816-2f3d-4007-b635-2a79923951f1" (UID: "4ff36816-2f3d-4007-b635-2a79923951f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:23.773864 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.773829    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-workload-certs" pod "4ff36816-2f3d-4007-b635-2a79923951f1" (UID: "4ff36816-2f3d-4007-b635-2a79923951f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:23.774226 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.774177    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ff36816-2f3d-4007-b635-2a79923951f1-istiod-ca-cert" pod "4ff36816-2f3d-4007-b635-2a79923951f1" (UID: "4ff36816-2f3d-4007-b635-2a79923951f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:23.775740 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.775689    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.775740 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.775717    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.775960 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.775742    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.775960 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.775763    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.775960 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.775777    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ff36816-2f3d-4007-b635-2a79923951f1-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.775960 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.775928    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff36816-2f3d-4007-b635-2a79923951f1-istio-token" pod "4ff36816-2f3d-4007-b635-2a79923951f1" (UID: "4ff36816-2f3d-4007-b635-2a79923951f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:23.776165 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.775977    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff36816-2f3d-4007-b635-2a79923951f1-kube-api-access-hhcr9" pod "4ff36816-2f3d-4007-b635-2a79923951f1" (UID: "4ff36816-2f3d-4007-b635-2a79923951f1"). InnerVolumeSpecName "kube-api-access-hhcr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:23.776278 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.776256    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ff36816-2f3d-4007-b635-2a79923951f1-istio-podinfo" pod "4ff36816-2f3d-4007-b635-2a79923951f1" (UID: "4ff36816-2f3d-4007-b635-2a79923951f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:09:23.776326 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.776301    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-istio-envoy" pod "4ff36816-2f3d-4007-b635-2a79923951f1" (UID: "4ff36816-2f3d-4007-b635-2a79923951f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:23.877206 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.877177    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhcr9\" (UniqueName: \"kubernetes.io/projected/4ff36816-2f3d-4007-b635-2a79923951f1-kube-api-access-hhcr9\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.877206 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.877202    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ff36816-2f3d-4007-b635-2a79923951f1-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.877206 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.877211    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ff36816-2f3d-4007-b635-2a79923951f1-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.877435 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:23.877219    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ff36816-2f3d-4007-b635-2a79923951f1-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:24.618346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:24.618318    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"
Oct 30 13:09:24.695018 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:24.694992    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"]
Oct 30 13:09:24.706115 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:24.706092    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jm56b"]
Oct 30 13:09:25.610465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:25.610429    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ff36816-2f3d-4007-b635-2a79923951f1" path="/var/lib/kubelet/pods/4ff36816-2f3d-4007-b635-2a79923951f1/volumes"
Oct 30 13:09:27.502420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:27.501748    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"]
Oct 30 13:09:27.502420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:27.502031    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl" podUID="eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" containerName="istio-proxy" containerID="cri-o://46625b6a15e0aa6951c1cacf17549253710ea07957122bf593aadbac350d8584" gracePeriod=30
Oct 30 13:09:28.013386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.013345    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"]
Oct 30 13:09:28.037439 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.014593    5837 cpu_manager.go:414] "containerMap: removing container" podUID="4ff36816-2f3d-4007-b635-2a79923951f1" containerName="istio-proxy"
Oct 30 13:09:28.044231 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.044202    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ff36816-2f3d-4007-b635-2a79923951f1" containerName="istio-proxy"
Oct 30 13:09:28.044525 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.044506    5837 cpu_manager.go:414] "containerMap: removing container" podUID="d6247fe5-c255-44b4-9186-c68883bef67b" containerName="revoke-keys"
Oct 30 13:09:28.044655 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.044643    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6247fe5-c255-44b4-9186-c68883bef67b" containerName="revoke-keys"
Oct 30 13:09:28.044939 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.044923    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6247fe5-c255-44b4-9186-c68883bef67b" containerName="revoke-keys"
Oct 30 13:09:28.045058 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.045047    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ff36816-2f3d-4007-b635-2a79923951f1" containerName="istio-proxy"
Oct 30 13:09:28.069427 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.064901    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"
Oct 30 13:09:28.073419 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.070290    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"]
Oct 30 13:09:28.097420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.093057    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:09:28.097420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.093791    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:09:28.097420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.093898    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:09:28.097420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.093991    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:09:28.281471 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.280614    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27b16fff-20b4-4038-8c99-bd0982b583c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc\" (UID: \"27b16fff-20b4-4038-8c99-bd0982b583c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"
Oct 30 13:09:28.297420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.295883    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqvnw\" (UniqueName: \"kubernetes.io/projected/27b16fff-20b4-4038-8c99-bd0982b583c9-kube-api-access-dqvnw\") pod \"maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc\" (UID: \"27b16fff-20b4-4038-8c99-bd0982b583c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"
Oct 30 13:09:28.297420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.296360    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27b16fff-20b4-4038-8c99-bd0982b583c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc\" (UID: \"27b16fff-20b4-4038-8c99-bd0982b583c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"
Oct 30 13:09:28.402196 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.402149    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27b16fff-20b4-4038-8c99-bd0982b583c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc\" (UID: \"27b16fff-20b4-4038-8c99-bd0982b583c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"
Oct 30 13:09:28.403040 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.403011    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvnw\" (UniqueName: \"kubernetes.io/projected/27b16fff-20b4-4038-8c99-bd0982b583c9-kube-api-access-dqvnw\") pod \"maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc\" (UID: \"27b16fff-20b4-4038-8c99-bd0982b583c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"
Oct 30 13:09:28.403289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.403270    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27b16fff-20b4-4038-8c99-bd0982b583c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc\" (UID: \"27b16fff-20b4-4038-8c99-bd0982b583c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"
Oct 30 13:09:28.424420 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.404275    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27b16fff-20b4-4038-8c99-bd0982b583c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc\" (UID: \"27b16fff-20b4-4038-8c99-bd0982b583c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"
Oct 30 13:09:28.428418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.425543    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27b16fff-20b4-4038-8c99-bd0982b583c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc\" (UID: \"27b16fff-20b4-4038-8c99-bd0982b583c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"
Oct 30 13:09:28.432416 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.432094    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqvnw\" (UniqueName: \"kubernetes.io/projected/27b16fff-20b4-4038-8c99-bd0982b583c9-kube-api-access-dqvnw\") pod \"maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc\" (UID: \"27b16fff-20b4-4038-8c99-bd0982b583c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"
Oct 30 13:09:28.613368 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:28.613285    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"
Oct 30 13:09:29.235903 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:29.235816    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"]
Oct 30 13:09:29.242163 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:09:29.242108    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b16fff_20b4_4038_8c99_bd0982b583c9.slice/crio-c654d33ec3e764fcc24313bc4d0abfbb0449a8f45a71698c97e875ceb24c8f1b WatchSource:0}: Error finding container c654d33ec3e764fcc24313bc4d0abfbb0449a8f45a71698c97e875ceb24c8f1b: Status 404 returned error can't find the container with id c654d33ec3e764fcc24313bc4d0abfbb0449a8f45a71698c97e875ceb24c8f1b
Oct 30 13:09:29.884280 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:29.884193    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc" event={"ID":"27b16fff-20b4-4038-8c99-bd0982b583c9","Type":"ContainerStarted","Data":"73df2f0f9e44111c784343a7174921e41a0f812c1a596bde5da67f6d11c1cee3"}
Oct 30 13:09:29.884280 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:29.884249    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc" event={"ID":"27b16fff-20b4-4038-8c99-bd0982b583c9","Type":"ContainerStarted","Data":"c654d33ec3e764fcc24313bc4d0abfbb0449a8f45a71698c97e875ceb24c8f1b"}
Oct 30 13:09:29.926786 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:29.926736    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc" podStartSLOduration=2.926721698 podStartE2EDuration="2.926721698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:29.922462062 +0000 UTC m=+1958.446746281" watchObservedRunningTime="2026-10-30 13:09:29.926721698 +0000 UTC m=+1958.451005917"
Oct 30 13:09:30.894275 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:30.894240    5837 generic.go:356] "Generic (PLEG): container finished" podID="27b16fff-20b4-4038-8c99-bd0982b583c9" containerID="73df2f0f9e44111c784343a7174921e41a0f812c1a596bde5da67f6d11c1cee3" exitCode=0
Oct 30 13:09:30.894702 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:30.894330    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc" event={"ID":"27b16fff-20b4-4038-8c99-bd0982b583c9","Type":"ContainerDied","Data":"73df2f0f9e44111c784343a7174921e41a0f812c1a596bde5da67f6d11c1cee3"}
Oct 30 13:09:32.046740 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.046718    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"
Oct 30 13:09:32.118348 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.118320    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b16fff-20b4-4038-8c99-bd0982b583c9-kube-api-access-dqvnw\" (UniqueName: \"kubernetes.io/projected/27b16fff-20b4-4038-8c99-bd0982b583c9-kube-api-access-dqvnw\") pod \"27b16fff-20b4-4038-8c99-bd0982b583c9\" (UID: \"27b16fff-20b4-4038-8c99-bd0982b583c9\") "
Oct 30 13:09:32.118531 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.118446    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27b16fff-20b4-4038-8c99-bd0982b583c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27b16fff-20b4-4038-8c99-bd0982b583c9-maas-api-xks-ca\") pod \"27b16fff-20b4-4038-8c99-bd0982b583c9\" (UID: \"27b16fff-20b4-4038-8c99-bd0982b583c9\") "
Oct 30 13:09:32.118578 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.118567    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27b16fff-20b4-4038-8c99-bd0982b583c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27b16fff-20b4-4038-8c99-bd0982b583c9-maas-api-openshift-ca\") pod \"27b16fff-20b4-4038-8c99-bd0982b583c9\" (UID: \"27b16fff-20b4-4038-8c99-bd0982b583c9\") "
Oct 30 13:09:32.118991 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.118959    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b16fff-20b4-4038-8c99-bd0982b583c9-maas-api-openshift-ca" pod "27b16fff-20b4-4038-8c99-bd0982b583c9" (UID: "27b16fff-20b4-4038-8c99-bd0982b583c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:32.119143 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.119088    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27b16fff-20b4-4038-8c99-bd0982b583c9-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:32.121188 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.121165    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b16fff-20b4-4038-8c99-bd0982b583c9-maas-api-xks-ca" pod "27b16fff-20b4-4038-8c99-bd0982b583c9" (UID: "27b16fff-20b4-4038-8c99-bd0982b583c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:32.121338 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.121185    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b16fff-20b4-4038-8c99-bd0982b583c9-kube-api-access-dqvnw" pod "27b16fff-20b4-4038-8c99-bd0982b583c9" (UID: "27b16fff-20b4-4038-8c99-bd0982b583c9"). InnerVolumeSpecName "kube-api-access-dqvnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:32.174270 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.174187    5837 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:09:32.174270 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.174251    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl" podUID="eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:09:32.220041 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.220003    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27b16fff-20b4-4038-8c99-bd0982b583c9-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:32.220041 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.220035    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqvnw\" (UniqueName: \"kubernetes.io/projected/27b16fff-20b4-4038-8c99-bd0982b583c9-kube-api-access-dqvnw\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:32.830137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.830109    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:09:32.911440 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.911334    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"
Oct 30 13:09:32.911440 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.911350    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc" event={"ID":"27b16fff-20b4-4038-8c99-bd0982b583c9","Type":"ContainerDied","Data":"c654d33ec3e764fcc24313bc4d0abfbb0449a8f45a71698c97e875ceb24c8f1b"}
Oct 30 13:09:32.911440 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.911434    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c654d33ec3e764fcc24313bc4d0abfbb0449a8f45a71698c97e875ceb24c8f1b"
Oct 30 13:09:32.913079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.913056    5837 generic.go:356] "Generic (PLEG): container finished" podID="eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" containerID="46625b6a15e0aa6951c1cacf17549253710ea07957122bf593aadbac350d8584" exitCode=0
Oct 30 13:09:32.913230 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.913112    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"
Oct 30 13:09:32.913230 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.913143    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl" event={"ID":"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5","Type":"ContainerDied","Data":"46625b6a15e0aa6951c1cacf17549253710ea07957122bf593aadbac350d8584"}
Oct 30 13:09:32.913230 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.913182    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl" event={"ID":"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5","Type":"ContainerDied","Data":"7779493986c964a1d0d90f3d43cb5842718fbd7076dfae0935b8814f720ddabb"}
Oct 30 13:09:32.913230 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.913202    5837 scope.go:122] "RemoveContainer" containerID="46625b6a15e0aa6951c1cacf17549253710ea07957122bf593aadbac350d8584"
Oct 30 13:09:32.927981 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.927955    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-kube-api-access-pd62k\" (UniqueName: \"kubernetes.io/projected/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-kube-api-access-pd62k\") pod \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") "
Oct 30 13:09:32.928099 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.928010    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-envoy\") pod \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") "
Oct 30 13:09:32.928099 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.928059    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-token\" (UniqueName: \"kubernetes.io/projected/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-token\") pod \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") "
Oct 30 13:09:32.928172 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.928106    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-podinfo\") pod \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") "
Oct 30 13:09:32.928172 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.928162    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-credential-socket\") pod \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") "
Oct 30 13:09:32.928257 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.928244    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-data\") pod \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") "
Oct 30 13:09:32.928340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.928310    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-workload-certs\") pod \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") "
Oct 30 13:09:32.928375 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.928360    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istiod-ca-cert\") pod \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") "
Oct 30 13:09:32.928461 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.928446    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-workload-socket\") pod \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\" (UID: \"eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5\") "
Oct 30 13:09:32.929153 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.929132    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-workload-socket" pod "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" (UID: "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:32.929546 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.929526    5837 scope.go:122] "RemoveContainer" containerID="46625b6a15e0aa6951c1cacf17549253710ea07957122bf593aadbac350d8584"
Oct 30 13:09:32.929646 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.929633    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-credential-socket" pod "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" (UID: "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:32.929962 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.929942    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-workload-certs" pod "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" (UID: "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:32.930049 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.930015    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istiod-ca-cert" pod "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" (UID: "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:32.930049 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.930026    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-data" pod "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" (UID: "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:32.930158 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:09:32.930062    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46625b6a15e0aa6951c1cacf17549253710ea07957122bf593aadbac350d8584\": container with ID starting with 46625b6a15e0aa6951c1cacf17549253710ea07957122bf593aadbac350d8584 not found: ID does not exist" containerID="46625b6a15e0aa6951c1cacf17549253710ea07957122bf593aadbac350d8584"
Oct 30 13:09:32.930158 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.930097    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46625b6a15e0aa6951c1cacf17549253710ea07957122bf593aadbac350d8584"} err="failed to get container status \"46625b6a15e0aa6951c1cacf17549253710ea07957122bf593aadbac350d8584\": rpc error: code = NotFound desc = could not find container \"46625b6a15e0aa6951c1cacf17549253710ea07957122bf593aadbac350d8584\": container with ID starting with 46625b6a15e0aa6951c1cacf17549253710ea07957122bf593aadbac350d8584 not found: ID does not exist"
Oct 30 13:09:32.931927 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.931904    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-envoy" pod "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" (UID: "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:32.931927 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.931915    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-kube-api-access-pd62k" pod "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" (UID: "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5"). InnerVolumeSpecName "kube-api-access-pd62k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:32.932038 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.931944    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-token" pod "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" (UID: "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:32.932154 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:32.932134    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-podinfo" pod "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" (UID: "eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:09:33.029896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:33.029870    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:33.029896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:33.029898    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:33.030087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:33.029912    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:33.030087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:33.029926    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd62k\" (UniqueName: \"kubernetes.io/projected/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-kube-api-access-pd62k\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:33.030087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:33.029940    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:33.030087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:33.029953    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:33.030087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:33.029965    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:33.030087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:33.029976    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:33.030087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:33.029988    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:33.257969 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:33.257897    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"]
Oct 30 13:09:33.269091 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:33.269062    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-8sfhl"]
Oct 30 13:09:33.605185 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:33.605155    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" path="/var/lib/kubelet/pods/eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5/volumes"
Oct 30 13:09:38.464299 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.464257    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"]
Oct 30 13:09:38.464830 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.464558    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f" podUID="d630a8a0-5896-415e-8c51-34bcd9486545" containerName="payload-processing" containerID="cri-o://4a7fc0047b284c3217df98aa5bcad6d6dd3251c9c74fd44e10a2cf17719e2250" gracePeriod=30
Oct 30 13:09:38.484646 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.484608    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"]
Oct 30 13:09:38.484896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.484849    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5" podUID="542ca25a-e199-4923-bd3f-b39860cb678c" containerName="payload-pre-processing" containerID="cri-o://9ff4f9f1fde9f73840f6efe0112e367d03399df124d051bb9f9279a3afae6fc3" gracePeriod=30
Oct 30 13:09:38.620413 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.617619    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"]
Oct 30 13:09:38.622709 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.622679    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f3cb8-tzcs5"]
Oct 30 13:09:38.871866 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.871495    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"
Oct 30 13:09:38.889648 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.889615    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"
Oct 30 13:09:38.955563 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.955444    5837 generic.go:356] "Generic (PLEG): container finished" podID="d630a8a0-5896-415e-8c51-34bcd9486545" containerID="4a7fc0047b284c3217df98aa5bcad6d6dd3251c9c74fd44e10a2cf17719e2250" exitCode=0
Oct 30 13:09:38.955960 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.955603    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f" event={"ID":"d630a8a0-5896-415e-8c51-34bcd9486545","Type":"ContainerDied","Data":"4a7fc0047b284c3217df98aa5bcad6d6dd3251c9c74fd44e10a2cf17719e2250"}
Oct 30 13:09:38.955960 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.955662    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f" event={"ID":"d630a8a0-5896-415e-8c51-34bcd9486545","Type":"ContainerDied","Data":"87df5f6c7fadcff8cbf4b021a7f92810c1c89ae5433c31a16232ad39bbcae7d2"}
Oct 30 13:09:38.955960 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.955700    5837 scope.go:122] "RemoveContainer" containerID="4a7fc0047b284c3217df98aa5bcad6d6dd3251c9c74fd44e10a2cf17719e2250"
Oct 30 13:09:38.955960 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.955872    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"
Oct 30 13:09:38.958927 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.958900    5837 generic.go:356] "Generic (PLEG): container finished" podID="542ca25a-e199-4923-bd3f-b39860cb678c" containerID="9ff4f9f1fde9f73840f6efe0112e367d03399df124d051bb9f9279a3afae6fc3" exitCode=0
Oct 30 13:09:38.959112 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.959091    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5" event={"ID":"542ca25a-e199-4923-bd3f-b39860cb678c","Type":"ContainerDied","Data":"9ff4f9f1fde9f73840f6efe0112e367d03399df124d051bb9f9279a3afae6fc3"}
Oct 30 13:09:38.959205 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.959146    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5" event={"ID":"542ca25a-e199-4923-bd3f-b39860cb678c","Type":"ContainerDied","Data":"ddc78ce4157441c2e2260dc8e04908a3521ee49ae0f8009a9e3986736b6b79a7"}
Oct 30 13:09:38.959339 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.959219    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"
Oct 30 13:09:38.973895 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.973868    5837 scope.go:122] "RemoveContainer" containerID="4a7fc0047b284c3217df98aa5bcad6d6dd3251c9c74fd44e10a2cf17719e2250"
Oct 30 13:09:38.974235 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:09:38.974209    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7fc0047b284c3217df98aa5bcad6d6dd3251c9c74fd44e10a2cf17719e2250\": container with ID starting with 4a7fc0047b284c3217df98aa5bcad6d6dd3251c9c74fd44e10a2cf17719e2250 not found: ID does not exist" containerID="4a7fc0047b284c3217df98aa5bcad6d6dd3251c9c74fd44e10a2cf17719e2250"
Oct 30 13:09:38.974319 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.974272    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7fc0047b284c3217df98aa5bcad6d6dd3251c9c74fd44e10a2cf17719e2250"} err="failed to get container status \"4a7fc0047b284c3217df98aa5bcad6d6dd3251c9c74fd44e10a2cf17719e2250\": rpc error: code = NotFound desc = could not find container \"4a7fc0047b284c3217df98aa5bcad6d6dd3251c9c74fd44e10a2cf17719e2250\": container with ID starting with 4a7fc0047b284c3217df98aa5bcad6d6dd3251c9c74fd44e10a2cf17719e2250 not found: ID does not exist"
Oct 30 13:09:38.974319 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.974296    5837 scope.go:122] "RemoveContainer" containerID="9ff4f9f1fde9f73840f6efe0112e367d03399df124d051bb9f9279a3afae6fc3"
Oct 30 13:09:38.986184 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.986160    5837 scope.go:122] "RemoveContainer" containerID="9ff4f9f1fde9f73840f6efe0112e367d03399df124d051bb9f9279a3afae6fc3"
Oct 30 13:09:38.986614 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:09:38.986590    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff4f9f1fde9f73840f6efe0112e367d03399df124d051bb9f9279a3afae6fc3\": container with ID starting with 9ff4f9f1fde9f73840f6efe0112e367d03399df124d051bb9f9279a3afae6fc3 not found: ID does not exist" containerID="9ff4f9f1fde9f73840f6efe0112e367d03399df124d051bb9f9279a3afae6fc3"
Oct 30 13:09:38.986705 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:38.986624    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff4f9f1fde9f73840f6efe0112e367d03399df124d051bb9f9279a3afae6fc3"} err="failed to get container status \"9ff4f9f1fde9f73840f6efe0112e367d03399df124d051bb9f9279a3afae6fc3\": rpc error: code = NotFound desc = could not find container \"9ff4f9f1fde9f73840f6efe0112e367d03399df124d051bb9f9279a3afae6fc3\": container with ID starting with 9ff4f9f1fde9f73840f6efe0112e367d03399df124d051bb9f9279a3afae6fc3 not found: ID does not exist"
Oct 30 13:09:39.008134 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.008060    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542ca25a-e199-4923-bd3f-b39860cb678c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542ca25a-e199-4923-bd3f-b39860cb678c-plugins-config-volume\") pod \"542ca25a-e199-4923-bd3f-b39860cb678c\" (UID: \"542ca25a-e199-4923-bd3f-b39860cb678c\") "
Oct 30 13:09:39.008134 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.008118    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d630a8a0-5896-415e-8c51-34bcd9486545-kube-api-access-wchgn\" (UniqueName: \"kubernetes.io/projected/d630a8a0-5896-415e-8c51-34bcd9486545-kube-api-access-wchgn\") pod \"d630a8a0-5896-415e-8c51-34bcd9486545\" (UID: \"d630a8a0-5896-415e-8c51-34bcd9486545\") "
Oct 30 13:09:39.008340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.008147    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542ca25a-e199-4923-bd3f-b39860cb678c-kube-api-access-5wbp8\" (UniqueName: \"kubernetes.io/projected/542ca25a-e199-4923-bd3f-b39860cb678c-kube-api-access-5wbp8\") pod \"542ca25a-e199-4923-bd3f-b39860cb678c\" (UID: \"542ca25a-e199-4923-bd3f-b39860cb678c\") "
Oct 30 13:09:39.008340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.008202    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d630a8a0-5896-415e-8c51-34bcd9486545-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d630a8a0-5896-415e-8c51-34bcd9486545-plugins-config-volume\") pod \"d630a8a0-5896-415e-8c51-34bcd9486545\" (UID: \"d630a8a0-5896-415e-8c51-34bcd9486545\") "
Oct 30 13:09:39.008569 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.008542    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542ca25a-e199-4923-bd3f-b39860cb678c-plugins-config-volume" pod "542ca25a-e199-4923-bd3f-b39860cb678c" (UID: "542ca25a-e199-4923-bd3f-b39860cb678c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:39.008701 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.008672    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d630a8a0-5896-415e-8c51-34bcd9486545-plugins-config-volume" pod "d630a8a0-5896-415e-8c51-34bcd9486545" (UID: "d630a8a0-5896-415e-8c51-34bcd9486545"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:39.008806 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.008788    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542ca25a-e199-4923-bd3f-b39860cb678c-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:39.008881 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.008809    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d630a8a0-5896-415e-8c51-34bcd9486545-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:39.010606 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.010586    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542ca25a-e199-4923-bd3f-b39860cb678c-kube-api-access-5wbp8" pod "542ca25a-e199-4923-bd3f-b39860cb678c" (UID: "542ca25a-e199-4923-bd3f-b39860cb678c"). InnerVolumeSpecName "kube-api-access-5wbp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:39.010830 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.010812    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d630a8a0-5896-415e-8c51-34bcd9486545-kube-api-access-wchgn" pod "d630a8a0-5896-415e-8c51-34bcd9486545" (UID: "d630a8a0-5896-415e-8c51-34bcd9486545"). InnerVolumeSpecName "kube-api-access-wchgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:39.109679 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.109640    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wchgn\" (UniqueName: \"kubernetes.io/projected/d630a8a0-5896-415e-8c51-34bcd9486545-kube-api-access-wchgn\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:39.109679 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.109671    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wbp8\" (UniqueName: \"kubernetes.io/projected/542ca25a-e199-4923-bd3f-b39860cb678c-kube-api-access-5wbp8\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:39.255513 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.255477    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"]
Oct 30 13:09:39.259346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.259283    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a81a48-hgftc"]
Oct 30 13:09:39.304139 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.304110    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"]
Oct 30 13:09:39.309651 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.309622    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a81a48-5bc4b674db-5t28f"]
Oct 30 13:09:39.375418 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.371099    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"]
Oct 30 13:09:39.401526 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.401444    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a81a48-69c4b6847-mqhw5"]
Oct 30 13:09:39.605754 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.605666    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27b16fff-20b4-4038-8c99-bd0982b583c9" path="/var/lib/kubelet/pods/27b16fff-20b4-4038-8c99-bd0982b583c9/volumes"
Oct 30 13:09:39.606155 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.606071    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="542ca25a-e199-4923-bd3f-b39860cb678c" path="/var/lib/kubelet/pods/542ca25a-e199-4923-bd3f-b39860cb678c/volumes"
Oct 30 13:09:39.606449 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.606436    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6247fe5-c255-44b4-9186-c68883bef67b" path="/var/lib/kubelet/pods/d6247fe5-c255-44b4-9186-c68883bef67b/volumes"
Oct 30 13:09:39.607042 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:39.607027    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d630a8a0-5896-415e-8c51-34bcd9486545" path="/var/lib/kubelet/pods/d630a8a0-5896-415e-8c51-34bcd9486545/volumes"
Oct 30 13:09:43.053485 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:43.053442    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"]
Oct 30 13:09:43.086524 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:43.086488    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39faa139-jnjpg"]
Oct 30 13:09:43.627331 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:43.627294    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f22bb1d-f0c7-436e-b0d2-810e9098e80b" path="/var/lib/kubelet/pods/9f22bb1d-f0c7-436e-b0d2-810e9098e80b/volumes"
Oct 30 13:09:45.094050 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:09:45.094002    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:09:49.432519 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.432484    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"]
Oct 30 13:09:49.432982 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.432970    5837 cpu_manager.go:414] "containerMap: removing container" podUID="542ca25a-e199-4923-bd3f-b39860cb678c" containerName="payload-pre-processing"
Oct 30 13:09:49.433031 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.432983    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542ca25a-e199-4923-bd3f-b39860cb678c" containerName="payload-pre-processing"
Oct 30 13:09:49.433031 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.432999    5837 cpu_manager.go:414] "containerMap: removing container" podUID="eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" containerName="istio-proxy"
Oct 30 13:09:49.433031 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.433003    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" containerName="istio-proxy"
Oct 30 13:09:49.433132 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.433033    5837 cpu_manager.go:414] "containerMap: removing container" podUID="d630a8a0-5896-415e-8c51-34bcd9486545" containerName="payload-processing"
Oct 30 13:09:49.433132 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.433038    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d630a8a0-5896-415e-8c51-34bcd9486545" containerName="payload-processing"
Oct 30 13:09:49.433132 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.433048    5837 cpu_manager.go:414] "containerMap: removing container" podUID="27b16fff-20b4-4038-8c99-bd0982b583c9" containerName="revoke-keys"
Oct 30 13:09:49.433132 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.433052    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27b16fff-20b4-4038-8c99-bd0982b583c9" containerName="revoke-keys"
Oct 30 13:09:49.433132 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.433129    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="542ca25a-e199-4923-bd3f-b39860cb678c" containerName="payload-pre-processing"
Oct 30 13:09:49.433296 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.433140    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8e4b82-23e0-4e40-a20a-1ec91fba2aa5" containerName="istio-proxy"
Oct 30 13:09:49.433296 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.433150    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d630a8a0-5896-415e-8c51-34bcd9486545" containerName="payload-processing"
Oct 30 13:09:49.433296 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.433157    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="27b16fff-20b4-4038-8c99-bd0982b583c9" containerName="revoke-keys"
Oct 30 13:09:49.436622 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.436605    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"
Oct 30 13:09:49.451425 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.451403    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:09:49.455605 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.455577    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"]
Oct 30 13:09:49.464349 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.462987    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:09:49.464349 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.463012    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:09:49.464349 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.463034    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:09:49.512436 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.512382    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b331e440-2ecf-4b68-ad58-d593926ad417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad83ceac-hljf7\" (UID: \"b331e440-2ecf-4b68-ad58-d593926ad417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"
Oct 30 13:09:49.512643 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.512527    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhgb5\" (UniqueName: \"kubernetes.io/projected/b331e440-2ecf-4b68-ad58-d593926ad417-kube-api-access-qhgb5\") pod \"maas-api-revoke-sub-ad83ceac-hljf7\" (UID: \"b331e440-2ecf-4b68-ad58-d593926ad417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"
Oct 30 13:09:49.512643 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.512584    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b331e440-2ecf-4b68-ad58-d593926ad417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad83ceac-hljf7\" (UID: \"b331e440-2ecf-4b68-ad58-d593926ad417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"
Oct 30 13:09:49.614090 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.614059    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b331e440-2ecf-4b68-ad58-d593926ad417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad83ceac-hljf7\" (UID: \"b331e440-2ecf-4b68-ad58-d593926ad417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"
Oct 30 13:09:49.614287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.614131    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhgb5\" (UniqueName: \"kubernetes.io/projected/b331e440-2ecf-4b68-ad58-d593926ad417-kube-api-access-qhgb5\") pod \"maas-api-revoke-sub-ad83ceac-hljf7\" (UID: \"b331e440-2ecf-4b68-ad58-d593926ad417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"
Oct 30 13:09:49.614287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.614160    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b331e440-2ecf-4b68-ad58-d593926ad417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad83ceac-hljf7\" (UID: \"b331e440-2ecf-4b68-ad58-d593926ad417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"
Oct 30 13:09:49.614864 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.614834    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b331e440-2ecf-4b68-ad58-d593926ad417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad83ceac-hljf7\" (UID: \"b331e440-2ecf-4b68-ad58-d593926ad417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"
Oct 30 13:09:49.616739 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.616711    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b331e440-2ecf-4b68-ad58-d593926ad417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad83ceac-hljf7\" (UID: \"b331e440-2ecf-4b68-ad58-d593926ad417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"
Oct 30 13:09:49.634896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.634873    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhgb5\" (UniqueName: \"kubernetes.io/projected/b331e440-2ecf-4b68-ad58-d593926ad417-kube-api-access-qhgb5\") pod \"maas-api-revoke-sub-ad83ceac-hljf7\" (UID: \"b331e440-2ecf-4b68-ad58-d593926ad417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"
Oct 30 13:09:49.750252 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:49.750219    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"
Oct 30 13:09:50.056442 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:50.056385    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"]
Oct 30 13:09:50.063202 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:09:50.063169    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb331e440_2ecf_4b68_ad58_d593926ad417.slice/crio-baf2f40aa29889bcdcd54e4f06646d5a23abf63a62eb59c3a8b10e8e6719de54 WatchSource:0}: Error finding container baf2f40aa29889bcdcd54e4f06646d5a23abf63a62eb59c3a8b10e8e6719de54: Status 404 returned error can't find the container with id baf2f40aa29889bcdcd54e4f06646d5a23abf63a62eb59c3a8b10e8e6719de54
Oct 30 13:09:50.132623 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:50.132584    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7" event={"ID":"b331e440-2ecf-4b68-ad58-d593926ad417","Type":"ContainerStarted","Data":"baf2f40aa29889bcdcd54e4f06646d5a23abf63a62eb59c3a8b10e8e6719de54"}
Oct 30 13:09:51.140571 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:51.140537    5837 generic.go:356] "Generic (PLEG): container finished" podID="b331e440-2ecf-4b68-ad58-d593926ad417" containerID="19a476163249faa27121d12a83d2b40d67a50e8274dc803fb2bb1b7e8f5e46a4" exitCode=0
Oct 30 13:09:51.140962 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:51.140574    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7" event={"ID":"b331e440-2ecf-4b68-ad58-d593926ad417","Type":"ContainerDied","Data":"19a476163249faa27121d12a83d2b40d67a50e8274dc803fb2bb1b7e8f5e46a4"}
Oct 30 13:09:52.825505 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:52.819499    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"
Oct 30 13:09:52.908426 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:52.896122    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b331e440-2ecf-4b68-ad58-d593926ad417-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b331e440-2ecf-4b68-ad58-d593926ad417-maas-api-xks-ca\") pod \"b331e440-2ecf-4b68-ad58-d593926ad417\" (UID: \"b331e440-2ecf-4b68-ad58-d593926ad417\") "
Oct 30 13:09:52.909257 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:52.909229    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b331e440-2ecf-4b68-ad58-d593926ad417-kube-api-access-qhgb5\" (UniqueName: \"kubernetes.io/projected/b331e440-2ecf-4b68-ad58-d593926ad417-kube-api-access-qhgb5\") pod \"b331e440-2ecf-4b68-ad58-d593926ad417\" (UID: \"b331e440-2ecf-4b68-ad58-d593926ad417\") "
Oct 30 13:09:53.039877 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:53.039839    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b331e440-2ecf-4b68-ad58-d593926ad417-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b331e440-2ecf-4b68-ad58-d593926ad417-maas-api-openshift-ca\") pod \"b331e440-2ecf-4b68-ad58-d593926ad417\" (UID: \"b331e440-2ecf-4b68-ad58-d593926ad417\") "
Oct 30 13:09:53.046609 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:53.046578    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b331e440-2ecf-4b68-ad58-d593926ad417-maas-api-openshift-ca" pod "b331e440-2ecf-4b68-ad58-d593926ad417" (UID: "b331e440-2ecf-4b68-ad58-d593926ad417"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:53.149537 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:53.149417    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b331e440-2ecf-4b68-ad58-d593926ad417-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:53.213421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:53.182280    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b331e440-2ecf-4b68-ad58-d593926ad417-maas-api-xks-ca" pod "b331e440-2ecf-4b68-ad58-d593926ad417" (UID: "b331e440-2ecf-4b68-ad58-d593926ad417"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:53.213421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:53.182954    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b331e440-2ecf-4b68-ad58-d593926ad417-kube-api-access-qhgb5" pod "b331e440-2ecf-4b68-ad58-d593926ad417" (UID: "b331e440-2ecf-4b68-ad58-d593926ad417"). InnerVolumeSpecName "kube-api-access-qhgb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:53.263087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:53.263047    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7" event={"ID":"b331e440-2ecf-4b68-ad58-d593926ad417","Type":"ContainerDied","Data":"baf2f40aa29889bcdcd54e4f06646d5a23abf63a62eb59c3a8b10e8e6719de54"}
Oct 30 13:09:53.263358 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:53.263339    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baf2f40aa29889bcdcd54e4f06646d5a23abf63a62eb59c3a8b10e8e6719de54"
Oct 30 13:09:53.277426 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:53.263617    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad83ceac-hljf7"
Oct 30 13:09:53.311351 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:53.311325    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhgb5\" (UniqueName: \"kubernetes.io/projected/b331e440-2ecf-4b68-ad58-d593926ad417-kube-api-access-qhgb5\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:53.311597 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:53.311581    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b331e440-2ecf-4b68-ad58-d593926ad417-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:09:58.044828 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:58.044781    5837 scope.go:122] "RemoveContainer" containerID="6369ed25c82ccd8c5ed6b22f4e73240fc38739b0121315da8a00750c1df93778"
Oct 30 13:09:58.058885 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:09:58.058858    5837 scope.go:122] "RemoveContainer" containerID="151e60e39d83fd2506c45e4d5739e742297a2df0e00982242665971576982106"
Oct 30 13:10:00.298421 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:00.297482    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:10:00.453994 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:00.453946    5837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:00.739550 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:00.739505    5837 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:10:00.739838 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:00.739807    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" podUID="25644850-37a6-4820-8ed2-4f00fedd8274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:01.205727 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.205630    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"]
Oct 30 13:10:01.206814 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.206787    5837 cpu_manager.go:414] "containerMap: removing container" podUID="b331e440-2ecf-4b68-ad58-d593926ad417" containerName="revoke-keys"
Oct 30 13:10:01.207299 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.207277    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b331e440-2ecf-4b68-ad58-d593926ad417" containerName="revoke-keys"
Oct 30 13:10:01.211428 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.207609    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b331e440-2ecf-4b68-ad58-d593926ad417" containerName="revoke-keys"
Oct 30 13:10:01.445645 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.445609    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"]
Oct 30 13:10:01.446329 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.446255    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"
Oct 30 13:10:01.468909 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.467418    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:10:01.470772 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.470726    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:10:01.481220 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.480594    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:10:01.489584 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.489563    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:10:01.560163 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.560122    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd8wg\" (UniqueName: \"kubernetes.io/projected/e3ad506d-54ac-4607-a246-f7a36984de73-kube-api-access-sd8wg\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67\" (UID: \"e3ad506d-54ac-4607-a246-f7a36984de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"
Oct 30 13:10:01.560566 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.560537    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ad506d-54ac-4607-a246-f7a36984de73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67\" (UID: \"e3ad506d-54ac-4607-a246-f7a36984de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"
Oct 30 13:10:01.560736 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.560715    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ad506d-54ac-4607-a246-f7a36984de73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67\" (UID: \"e3ad506d-54ac-4607-a246-f7a36984de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"
Oct 30 13:10:01.662614 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.662580    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd8wg\" (UniqueName: \"kubernetes.io/projected/e3ad506d-54ac-4607-a246-f7a36984de73-kube-api-access-sd8wg\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67\" (UID: \"e3ad506d-54ac-4607-a246-f7a36984de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"
Oct 30 13:10:01.662842 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.662718    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ad506d-54ac-4607-a246-f7a36984de73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67\" (UID: \"e3ad506d-54ac-4607-a246-f7a36984de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"
Oct 30 13:10:01.662842 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.662750    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ad506d-54ac-4607-a246-f7a36984de73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67\" (UID: \"e3ad506d-54ac-4607-a246-f7a36984de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"
Oct 30 13:10:01.663500 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.663478    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ad506d-54ac-4607-a246-f7a36984de73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67\" (UID: \"e3ad506d-54ac-4607-a246-f7a36984de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"
Oct 30 13:10:01.665718 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.665692    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ad506d-54ac-4607-a246-f7a36984de73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67\" (UID: \"e3ad506d-54ac-4607-a246-f7a36984de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"
Oct 30 13:10:01.681874 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.681840    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd8wg\" (UniqueName: \"kubernetes.io/projected/e3ad506d-54ac-4607-a246-f7a36984de73-kube-api-access-sd8wg\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67\" (UID: \"e3ad506d-54ac-4607-a246-f7a36984de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"
Oct 30 13:10:01.796937 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:01.796867    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"
Oct 30 13:10:02.096907 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:02.096878    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"]
Oct 30 13:10:02.097630 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:10:02.097604    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ad506d_54ac_4607_a246_f7a36984de73.slice/crio-adc9aba3a2e2f758494c029a609495d432baa784595e873dc3f100bac3dcc2e7 WatchSource:0}: Error finding container adc9aba3a2e2f758494c029a609495d432baa784595e873dc3f100bac3dcc2e7: Status 404 returned error can't find the container with id adc9aba3a2e2f758494c029a609495d432baa784595e873dc3f100bac3dcc2e7
Oct 30 13:10:02.473471 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:02.473431    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67" event={"ID":"e3ad506d-54ac-4607-a246-f7a36984de73","Type":"ContainerStarted","Data":"791db7e82050314afc88e4ad62affbcdb00acf6f9ad6b47d51f1661b6f45b034"}
Oct 30 13:10:02.473983 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:02.473502    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67" event={"ID":"e3ad506d-54ac-4607-a246-f7a36984de73","Type":"ContainerStarted","Data":"adc9aba3a2e2f758494c029a609495d432baa784595e873dc3f100bac3dcc2e7"}
Oct 30 13:10:02.512689 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:02.512636    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67" podStartSLOduration=1.512619545 podStartE2EDuration="1.512619545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:02.510569801 +0000 UTC m=+1991.034854019" watchObservedRunningTime="2026-10-30 13:10:02.512619545 +0000 UTC m=+1991.036903764"
Oct 30 13:10:03.482225 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:03.482192    5837 generic.go:356] "Generic (PLEG): container finished" podID="e3ad506d-54ac-4607-a246-f7a36984de73" containerID="791db7e82050314afc88e4ad62affbcdb00acf6f9ad6b47d51f1661b6f45b034" exitCode=0
Oct 30 13:10:03.482639 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:03.482278    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67" event={"ID":"e3ad506d-54ac-4607-a246-f7a36984de73","Type":"ContainerDied","Data":"791db7e82050314afc88e4ad62affbcdb00acf6f9ad6b47d51f1661b6f45b034"}
Oct 30 13:10:04.661231 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:04.661210    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"
Oct 30 13:10:04.798598 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:04.798514    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ad506d-54ac-4607-a246-f7a36984de73-kube-api-access-sd8wg\" (UniqueName: \"kubernetes.io/projected/e3ad506d-54ac-4607-a246-f7a36984de73-kube-api-access-sd8wg\") pod \"e3ad506d-54ac-4607-a246-f7a36984de73\" (UID: \"e3ad506d-54ac-4607-a246-f7a36984de73\") "
Oct 30 13:10:04.798753 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:04.798644    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3ad506d-54ac-4607-a246-f7a36984de73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ad506d-54ac-4607-a246-f7a36984de73-maas-api-xks-ca\") pod \"e3ad506d-54ac-4607-a246-f7a36984de73\" (UID: \"e3ad506d-54ac-4607-a246-f7a36984de73\") "
Oct 30 13:10:04.798753 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:04.798686    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ad506d-54ac-4607-a246-f7a36984de73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ad506d-54ac-4607-a246-f7a36984de73-maas-api-openshift-ca\") pod \"e3ad506d-54ac-4607-a246-f7a36984de73\" (UID: \"e3ad506d-54ac-4607-a246-f7a36984de73\") "
Oct 30 13:10:04.799079 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:04.799055    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ad506d-54ac-4607-a246-f7a36984de73-maas-api-openshift-ca" pod "e3ad506d-54ac-4607-a246-f7a36984de73" (UID: "e3ad506d-54ac-4607-a246-f7a36984de73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:04.799343 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:04.799170    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ad506d-54ac-4607-a246-f7a36984de73-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:04.800811 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:04.800787    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ad506d-54ac-4607-a246-f7a36984de73-kube-api-access-sd8wg" pod "e3ad506d-54ac-4607-a246-f7a36984de73" (UID: "e3ad506d-54ac-4607-a246-f7a36984de73"). InnerVolumeSpecName "kube-api-access-sd8wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:04.800911 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:04.800867    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ad506d-54ac-4607-a246-f7a36984de73-maas-api-xks-ca" pod "e3ad506d-54ac-4607-a246-f7a36984de73" (UID: "e3ad506d-54ac-4607-a246-f7a36984de73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:04.900049 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:04.900015    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd8wg\" (UniqueName: \"kubernetes.io/projected/e3ad506d-54ac-4607-a246-f7a36984de73-kube-api-access-sd8wg\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:04.900049 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:04.900051    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ad506d-54ac-4607-a246-f7a36984de73-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:05.497030 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:05.497000    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"
Oct 30 13:10:05.497204 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:05.497032    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67" event={"ID":"e3ad506d-54ac-4607-a246-f7a36984de73","Type":"ContainerDied","Data":"adc9aba3a2e2f758494c029a609495d432baa784595e873dc3f100bac3dcc2e7"}
Oct 30 13:10:05.497204 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:05.497112    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adc9aba3a2e2f758494c029a609495d432baa784595e873dc3f100bac3dcc2e7"
Oct 30 13:10:05.913801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:05.913727    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"]
Oct 30 13:10:06.467593 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:06.467560    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"]
Oct 30 13:10:11.621347 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:11.621313    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"]
Oct 30 13:10:11.652022 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:11.651991    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"]
Oct 30 13:10:12.345984 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:12.345946    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"]
Oct 30 13:10:12.370936 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:12.370902    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"]
Oct 30 13:10:13.834572 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.834543    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"]
Oct 30 13:10:13.835072 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.835059    5837 cpu_manager.go:414] "containerMap: removing container" podUID="e3ad506d-54ac-4607-a246-f7a36984de73" containerName="revoke-keys"
Oct 30 13:10:13.835120 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.835073    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ad506d-54ac-4607-a246-f7a36984de73" containerName="revoke-keys"
Oct 30 13:10:13.835167 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.835158    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ad506d-54ac-4607-a246-f7a36984de73" containerName="revoke-keys"
Oct 30 13:10:13.838262 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.838244    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"
Oct 30 13:10:13.853410 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.853379    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:10:13.853553 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.853508    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:10:13.856367 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.856353    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:10:13.865703 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.865685    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:10:13.868908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.868884    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"]
Oct 30 13:10:13.928287 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.928259    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4348ac37-952a-4000-ba42-3a8fa6f12c4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b2fb3a2-g8c2p\" (UID: \"4348ac37-952a-4000-ba42-3a8fa6f12c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"
Oct 30 13:10:13.928503 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.928313    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t99zc\" (UniqueName: \"kubernetes.io/projected/4348ac37-952a-4000-ba42-3a8fa6f12c4d-kube-api-access-t99zc\") pod \"maas-api-revoke-sub-5b2fb3a2-g8c2p\" (UID: \"4348ac37-952a-4000-ba42-3a8fa6f12c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"
Oct 30 13:10:13.928503 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.928423    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4348ac37-952a-4000-ba42-3a8fa6f12c4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b2fb3a2-g8c2p\" (UID: \"4348ac37-952a-4000-ba42-3a8fa6f12c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"
Oct 30 13:10:13.928892 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.928864    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"]
Oct 30 13:10:13.955189 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.955156    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"
Oct 30 13:10:13.956235 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:13.956209    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"]
Oct 30 13:10:14.030003 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.029966    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4348ac37-952a-4000-ba42-3a8fa6f12c4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b2fb3a2-g8c2p\" (UID: \"4348ac37-952a-4000-ba42-3a8fa6f12c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"
Oct 30 13:10:14.030181 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.030027    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4k2s\" (UniqueName: \"kubernetes.io/projected/7b0047c7-c493-412a-83e1-10b48bdc06b6-kube-api-access-w4k2s\") pod \"maas-api-revoke-sub-41ce7966-jd7mq\" (UID: \"7b0047c7-c493-412a-83e1-10b48bdc06b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"
Oct 30 13:10:14.030181 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.030140    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99zc\" (UniqueName: \"kubernetes.io/projected/4348ac37-952a-4000-ba42-3a8fa6f12c4d-kube-api-access-t99zc\") pod \"maas-api-revoke-sub-5b2fb3a2-g8c2p\" (UID: \"4348ac37-952a-4000-ba42-3a8fa6f12c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"
Oct 30 13:10:14.030181 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.030167    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0047c7-c493-412a-83e1-10b48bdc06b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41ce7966-jd7mq\" (UID: \"7b0047c7-c493-412a-83e1-10b48bdc06b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"
Oct 30 13:10:14.030294 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.030184    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4348ac37-952a-4000-ba42-3a8fa6f12c4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b2fb3a2-g8c2p\" (UID: \"4348ac37-952a-4000-ba42-3a8fa6f12c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"
Oct 30 13:10:14.030364 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.030342    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0047c7-c493-412a-83e1-10b48bdc06b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41ce7966-jd7mq\" (UID: \"7b0047c7-c493-412a-83e1-10b48bdc06b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"
Oct 30 13:10:14.030625 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.030608    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4348ac37-952a-4000-ba42-3a8fa6f12c4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b2fb3a2-g8c2p\" (UID: \"4348ac37-952a-4000-ba42-3a8fa6f12c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"
Oct 30 13:10:14.032939 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.032919    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4348ac37-952a-4000-ba42-3a8fa6f12c4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b2fb3a2-g8c2p\" (UID: \"4348ac37-952a-4000-ba42-3a8fa6f12c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"
Oct 30 13:10:14.048912 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.048886    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t99zc\" (UniqueName: \"kubernetes.io/projected/4348ac37-952a-4000-ba42-3a8fa6f12c4d-kube-api-access-t99zc\") pod \"maas-api-revoke-sub-5b2fb3a2-g8c2p\" (UID: \"4348ac37-952a-4000-ba42-3a8fa6f12c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"
Oct 30 13:10:14.131480 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.131334    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0047c7-c493-412a-83e1-10b48bdc06b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41ce7966-jd7mq\" (UID: \"7b0047c7-c493-412a-83e1-10b48bdc06b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"
Oct 30 13:10:14.131679 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.131507    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4k2s\" (UniqueName: \"kubernetes.io/projected/7b0047c7-c493-412a-83e1-10b48bdc06b6-kube-api-access-w4k2s\") pod \"maas-api-revoke-sub-41ce7966-jd7mq\" (UID: \"7b0047c7-c493-412a-83e1-10b48bdc06b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"
Oct 30 13:10:14.131679 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.131561    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0047c7-c493-412a-83e1-10b48bdc06b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41ce7966-jd7mq\" (UID: \"7b0047c7-c493-412a-83e1-10b48bdc06b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"
Oct 30 13:10:14.132206 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.132182    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0047c7-c493-412a-83e1-10b48bdc06b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41ce7966-jd7mq\" (UID: \"7b0047c7-c493-412a-83e1-10b48bdc06b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"
Oct 30 13:10:14.133873 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.133848    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0047c7-c493-412a-83e1-10b48bdc06b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41ce7966-jd7mq\" (UID: \"7b0047c7-c493-412a-83e1-10b48bdc06b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"
Oct 30 13:10:14.150895 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.150872    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4k2s\" (UniqueName: \"kubernetes.io/projected/7b0047c7-c493-412a-83e1-10b48bdc06b6-kube-api-access-w4k2s\") pod \"maas-api-revoke-sub-41ce7966-jd7mq\" (UID: \"7b0047c7-c493-412a-83e1-10b48bdc06b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"
Oct 30 13:10:14.151710 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.151689    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"
Oct 30 13:10:14.271059 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.271012    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"
Oct 30 13:10:14.447882 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.447849    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"]
Oct 30 13:10:14.448882 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:10:14.448854    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4348ac37_952a_4000_ba42_3a8fa6f12c4d.slice/crio-8b376c754e29fddac9b9e226c46f14da9db96ae96d1a44f40f6a150b9311e85c WatchSource:0}: Error finding container 8b376c754e29fddac9b9e226c46f14da9db96ae96d1a44f40f6a150b9311e85c: Status 404 returned error can't find the container with id 8b376c754e29fddac9b9e226c46f14da9db96ae96d1a44f40f6a150b9311e85c
Oct 30 13:10:14.567665 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.567641    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"]
Oct 30 13:10:14.569325 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:10:14.569296    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0047c7_c493_412a_83e1_10b48bdc06b6.slice/crio-d95aac88390f9d914d4a87fb6076a2e672fb2a084f79b057d1e570e44f3e88bc WatchSource:0}: Error finding container d95aac88390f9d914d4a87fb6076a2e672fb2a084f79b057d1e570e44f3e88bc: Status 404 returned error can't find the container with id d95aac88390f9d914d4a87fb6076a2e672fb2a084f79b057d1e570e44f3e88bc
Oct 30 13:10:14.741902 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.741864    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq" event={"ID":"7b0047c7-c493-412a-83e1-10b48bdc06b6","Type":"ContainerStarted","Data":"d95aac88390f9d914d4a87fb6076a2e672fb2a084f79b057d1e570e44f3e88bc"}
Oct 30 13:10:14.743469 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.743441    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p" event={"ID":"4348ac37-952a-4000-ba42-3a8fa6f12c4d","Type":"ContainerStarted","Data":"3cd1a3e3a10cf81c110826c2350c73a9740fb892df0cd5e8e4bf4dbc91438e1b"}
Oct 30 13:10:14.743604 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.743499    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p" event={"ID":"4348ac37-952a-4000-ba42-3a8fa6f12c4d","Type":"ContainerStarted","Data":"8b376c754e29fddac9b9e226c46f14da9db96ae96d1a44f40f6a150b9311e85c"}
Oct 30 13:10:14.784214 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:14.784163    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p" podStartSLOduration=1.784147782 podStartE2EDuration="1.784147782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:14.781893022 +0000 UTC m=+2003.306177241" watchObservedRunningTime="2026-10-30 13:10:14.784147782 +0000 UTC m=+2003.308432041"
Oct 30 13:10:15.070309 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:15.070276    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"]
Oct 30 13:10:15.074475 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:15.074450    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3598d2e5-n22nk"]
Oct 30 13:10:15.097451 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:15.097382    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:10:15.604747 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:15.604713    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="587f599f-eac3-4046-926f-9ee4a4f391e4" path="/var/lib/kubelet/pods/587f599f-eac3-4046-926f-9ee4a4f391e4/volumes"
Oct 30 13:10:15.751153 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:15.751119    5837 generic.go:356] "Generic (PLEG): container finished" podID="7b0047c7-c493-412a-83e1-10b48bdc06b6" containerID="1794fe1dca80581aba65006b18bc40d41a29c11e6475d6b9403f5fefe67877a9" exitCode=0
Oct 30 13:10:15.751348 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:15.751192    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq" event={"ID":"7b0047c7-c493-412a-83e1-10b48bdc06b6","Type":"ContainerDied","Data":"1794fe1dca80581aba65006b18bc40d41a29c11e6475d6b9403f5fefe67877a9"}
Oct 30 13:10:15.752805 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:15.752780    5837 generic.go:356] "Generic (PLEG): container finished" podID="4348ac37-952a-4000-ba42-3a8fa6f12c4d" containerID="3cd1a3e3a10cf81c110826c2350c73a9740fb892df0cd5e8e4bf4dbc91438e1b" exitCode=0
Oct 30 13:10:15.752927 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:15.752840    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p" event={"ID":"4348ac37-952a-4000-ba42-3a8fa6f12c4d","Type":"ContainerDied","Data":"3cd1a3e3a10cf81c110826c2350c73a9740fb892df0cd5e8e4bf4dbc91438e1b"}
Oct 30 13:10:16.030165 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.030130    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"]
Oct 30 13:10:16.032771 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.032748    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7824bafb-5wc5n"]
Oct 30 13:10:16.621713 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.621677    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv" podUID="3929deb3-fa21-4e2f-8ada-6348c9faac3d" containerName="payload-processing" containerID="cri-o://79ce72ddf662e4084f6df23a705dba8e1653abdcb6b252feb3d5dcf026fb666f" gracePeriod=25
Oct 30 13:10:16.652673 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.652635    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv" podUID="fb963087-5281-48f9-b322-110ae1590936" containerName="payload-pre-processing" containerID="cri-o://c1fe7e5a94a10a38b594db3ef4fb013beadda9db2b247ca4757e96cb6d774ed6" gracePeriod=25
Oct 30 13:10:16.938908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.938883    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"
Oct 30 13:10:16.941572 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.941551    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"
Oct 30 13:10:16.962433 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.962378    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b0047c7-c493-412a-83e1-10b48bdc06b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0047c7-c493-412a-83e1-10b48bdc06b6-maas-api-xks-ca\") pod \"7b0047c7-c493-412a-83e1-10b48bdc06b6\" (UID: \"7b0047c7-c493-412a-83e1-10b48bdc06b6\") "
Oct 30 13:10:16.962595 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.962489    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0047c7-c493-412a-83e1-10b48bdc06b6-kube-api-access-w4k2s\" (UniqueName: \"kubernetes.io/projected/7b0047c7-c493-412a-83e1-10b48bdc06b6-kube-api-access-w4k2s\") pod \"7b0047c7-c493-412a-83e1-10b48bdc06b6\" (UID: \"7b0047c7-c493-412a-83e1-10b48bdc06b6\") "
Oct 30 13:10:16.962595 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.962539    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4348ac37-952a-4000-ba42-3a8fa6f12c4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4348ac37-952a-4000-ba42-3a8fa6f12c4d-maas-api-xks-ca\") pod \"4348ac37-952a-4000-ba42-3a8fa6f12c4d\" (UID: \"4348ac37-952a-4000-ba42-3a8fa6f12c4d\") "
Oct 30 13:10:16.962595 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.962580    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4348ac37-952a-4000-ba42-3a8fa6f12c4d-kube-api-access-t99zc\" (UniqueName: \"kubernetes.io/projected/4348ac37-952a-4000-ba42-3a8fa6f12c4d-kube-api-access-t99zc\") pod \"4348ac37-952a-4000-ba42-3a8fa6f12c4d\" (UID: \"4348ac37-952a-4000-ba42-3a8fa6f12c4d\") "
Oct 30 13:10:16.962776 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.962653    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4348ac37-952a-4000-ba42-3a8fa6f12c4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4348ac37-952a-4000-ba42-3a8fa6f12c4d-maas-api-openshift-ca\") pod \"4348ac37-952a-4000-ba42-3a8fa6f12c4d\" (UID: \"4348ac37-952a-4000-ba42-3a8fa6f12c4d\") "
Oct 30 13:10:16.962776 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.962695    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0047c7-c493-412a-83e1-10b48bdc06b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0047c7-c493-412a-83e1-10b48bdc06b6-maas-api-openshift-ca\") pod \"7b0047c7-c493-412a-83e1-10b48bdc06b6\" (UID: \"7b0047c7-c493-412a-83e1-10b48bdc06b6\") "
Oct 30 13:10:16.963115 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.963033    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4348ac37-952a-4000-ba42-3a8fa6f12c4d-maas-api-openshift-ca" pod "4348ac37-952a-4000-ba42-3a8fa6f12c4d" (UID: "4348ac37-952a-4000-ba42-3a8fa6f12c4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:16.963115 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.963119    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0047c7-c493-412a-83e1-10b48bdc06b6-maas-api-openshift-ca" pod "7b0047c7-c493-412a-83e1-10b48bdc06b6" (UID: "7b0047c7-c493-412a-83e1-10b48bdc06b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:16.964803 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.964767    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4348ac37-952a-4000-ba42-3a8fa6f12c4d-maas-api-xks-ca" pod "4348ac37-952a-4000-ba42-3a8fa6f12c4d" (UID: "4348ac37-952a-4000-ba42-3a8fa6f12c4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:16.964921 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.964807    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0047c7-c493-412a-83e1-10b48bdc06b6-maas-api-xks-ca" pod "7b0047c7-c493-412a-83e1-10b48bdc06b6" (UID: "7b0047c7-c493-412a-83e1-10b48bdc06b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:16.964989 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.964924    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0047c7-c493-412a-83e1-10b48bdc06b6-kube-api-access-w4k2s" pod "7b0047c7-c493-412a-83e1-10b48bdc06b6" (UID: "7b0047c7-c493-412a-83e1-10b48bdc06b6"). InnerVolumeSpecName "kube-api-access-w4k2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:16.964989 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:16.964957    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4348ac37-952a-4000-ba42-3a8fa6f12c4d-kube-api-access-t99zc" pod "4348ac37-952a-4000-ba42-3a8fa6f12c4d" (UID: "4348ac37-952a-4000-ba42-3a8fa6f12c4d"). InnerVolumeSpecName "kube-api-access-t99zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:17.064316 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:17.064275    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4k2s\" (UniqueName: \"kubernetes.io/projected/7b0047c7-c493-412a-83e1-10b48bdc06b6-kube-api-access-w4k2s\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.064316 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:17.064321    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4348ac37-952a-4000-ba42-3a8fa6f12c4d-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.064543 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:17.064336    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t99zc\" (UniqueName: \"kubernetes.io/projected/4348ac37-952a-4000-ba42-3a8fa6f12c4d-kube-api-access-t99zc\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.064543 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:17.064350    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4348ac37-952a-4000-ba42-3a8fa6f12c4d-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.064543 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:17.064364    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0047c7-c493-412a-83e1-10b48bdc06b6-maas-api-openshift-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.064543 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:17.064378    5837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0047c7-c493-412a-83e1-10b48bdc06b6-maas-api-xks-ca\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.346742 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:17.346641    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q" podUID="bc1998d0-b85e-46aa-a6d4-92b828b66d08" containerName="payload-processing" containerID="cri-o://ac7b1dbb3c16a1bfe80cde0e1b6a64c91a979b62fb7df37c5173b036c2fee1db" gracePeriod=25
Oct 30 13:10:17.371374 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:17.371330    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9" podUID="ee36af90-164d-404c-be97-5dbaeafc0648" containerName="payload-pre-processing" containerID="cri-o://155294b2dc20a7b61ec74a3be78fd3cdbd112919afc821e8ff008f839b3276b8" gracePeriod=25
Oct 30 13:10:17.605340 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:17.605246    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c59a19-6c67-42f0-9703-7dbf1d4b6489" path="/var/lib/kubelet/pods/b2c59a19-6c67-42f0-9703-7dbf1d4b6489/volumes"
Oct 30 13:10:17.768467 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:17.768439    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq"
Oct 30 13:10:17.768901 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:17.768431    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ce7966-jd7mq" event={"ID":"7b0047c7-c493-412a-83e1-10b48bdc06b6","Type":"ContainerDied","Data":"d95aac88390f9d914d4a87fb6076a2e672fb2a084f79b057d1e570e44f3e88bc"}
Oct 30 13:10:17.768901 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:17.768598    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d95aac88390f9d914d4a87fb6076a2e672fb2a084f79b057d1e570e44f3e88bc"
Oct 30 13:10:17.769962 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:17.769935    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p" event={"ID":"4348ac37-952a-4000-ba42-3a8fa6f12c4d","Type":"ContainerDied","Data":"8b376c754e29fddac9b9e226c46f14da9db96ae96d1a44f40f6a150b9311e85c"}
Oct 30 13:10:17.770123 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:17.769967    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2fb3a2-g8c2p"
Oct 30 13:10:17.770123 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:17.769984    5837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b376c754e29fddac9b9e226c46f14da9db96ae96d1a44f40f6a150b9311e85c"
Oct 30 13:10:20.891167 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:20.891126    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv" podUID="3929deb3-fa21-4e2f-8ada-6348c9faac3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:10:20.891695 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:20.891137    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv" podUID="fb963087-5281-48f9-b322-110ae1590936" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:10:20.935348 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:20.935289    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main" containerID="cri-o://01141dc5895af7f8abb7e47c9a0fdecd21088727db28a51324ede210d5af9281" gracePeriod=45
Oct 30 13:10:21.208070 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.208036    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:10:21.313329 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.313290    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6eaa6ea-8274-49fa-b007-753de046e268-kube-api-access-lt5lj\" (UniqueName: \"kubernetes.io/projected/d6eaa6ea-8274-49fa-b007-753de046e268-kube-api-access-lt5lj\") pod \"d6eaa6ea-8274-49fa-b007-753de046e268\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") "
Oct 30 13:10:21.313546 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.313477    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6eaa6ea-8274-49fa-b007-753de046e268-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6eaa6ea-8274-49fa-b007-753de046e268-tls-certs\") pod \"d6eaa6ea-8274-49fa-b007-753de046e268\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") "
Oct 30 13:10:21.313546 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.313509    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-dshm\") pod \"d6eaa6ea-8274-49fa-b007-753de046e268\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") "
Oct 30 13:10:21.313669 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.313597    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-home\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-home\") pod \"d6eaa6ea-8274-49fa-b007-753de046e268\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") "
Oct 30 13:10:21.313726 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.313693    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-model-cache\") pod \"d6eaa6ea-8274-49fa-b007-753de046e268\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") "
Oct 30 13:10:21.313810 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.313792    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-tmp-dir\") pod \"d6eaa6ea-8274-49fa-b007-753de046e268\" (UID: \"d6eaa6ea-8274-49fa-b007-753de046e268\") "
Oct 30 13:10:21.313923 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.313894    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-home" pod "d6eaa6ea-8274-49fa-b007-753de046e268" (UID: "d6eaa6ea-8274-49fa-b007-753de046e268"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:21.314068 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.314040    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-model-cache" pod "d6eaa6ea-8274-49fa-b007-753de046e268" (UID: "d6eaa6ea-8274-49fa-b007-753de046e268"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:21.314233 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.314210    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-tmp-dir" pod "d6eaa6ea-8274-49fa-b007-753de046e268" (UID: "d6eaa6ea-8274-49fa-b007-753de046e268"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:21.314590 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.314561    5837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-home\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:21.314590 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.314590    5837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-model-cache\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:21.314808 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.314606    5837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-tmp-dir\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:21.319837 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.319809    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-dshm" pod "d6eaa6ea-8274-49fa-b007-753de046e268" (UID: "d6eaa6ea-8274-49fa-b007-753de046e268"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:21.319948 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.319931    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6eaa6ea-8274-49fa-b007-753de046e268-kube-api-access-lt5lj" pod "d6eaa6ea-8274-49fa-b007-753de046e268" (UID: "d6eaa6ea-8274-49fa-b007-753de046e268"). InnerVolumeSpecName "kube-api-access-lt5lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:21.327556 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.327518    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6eaa6ea-8274-49fa-b007-753de046e268-tls-certs" pod "d6eaa6ea-8274-49fa-b007-753de046e268" (UID: "d6eaa6ea-8274-49fa-b007-753de046e268"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:21.419251 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.419215    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt5lj\" (UniqueName: \"kubernetes.io/projected/d6eaa6ea-8274-49fa-b007-753de046e268-kube-api-access-lt5lj\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:21.419251 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.419254    5837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6eaa6ea-8274-49fa-b007-753de046e268-tls-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:21.419526 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.419269    5837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6eaa6ea-8274-49fa-b007-753de046e268-dshm\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:21.489063 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.488998    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" containerID="cri-o://4175e65a2184a1c266ff1f960db7f72fcd203e4b584fd92fc131624abe642770" gracePeriod=45
Oct 30 13:10:21.612157 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.612129    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": dial tcp 10.134.0.100:8000: connect: connection refused"
Oct 30 13:10:21.840875 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.840571    5837 generic.go:356] "Generic (PLEG): container finished" podID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerID="4175e65a2184a1c266ff1f960db7f72fcd203e4b584fd92fc131624abe642770" exitCode=0
Oct 30 13:10:21.840875 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.840756    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" event={"ID":"089cb853-6545-4a2e-9aed-3b6cede912d6","Type":"ContainerDied","Data":"4175e65a2184a1c266ff1f960db7f72fcd203e4b584fd92fc131624abe642770"}
Oct 30 13:10:21.844259 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.842896    5837 generic.go:356] "Generic (PLEG): container finished" podID="d6eaa6ea-8274-49fa-b007-753de046e268" containerID="01141dc5895af7f8abb7e47c9a0fdecd21088727db28a51324ede210d5af9281" exitCode=0
Oct 30 13:10:21.844259 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.843101    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" event={"ID":"d6eaa6ea-8274-49fa-b007-753de046e268","Type":"ContainerDied","Data":"01141dc5895af7f8abb7e47c9a0fdecd21088727db28a51324ede210d5af9281"}
Oct 30 13:10:21.844259 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.843132    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762" event={"ID":"d6eaa6ea-8274-49fa-b007-753de046e268","Type":"ContainerDied","Data":"82c8aff0a04d9661da73c494f6d62824a4260f499e42c30ca89938a0d7637918"}
Oct 30 13:10:21.844259 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.843152    5837 scope.go:122] "RemoveContainer" containerID="01141dc5895af7f8abb7e47c9a0fdecd21088727db28a51324ede210d5af9281"
Oct 30 13:10:21.844259 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.843306    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"
Oct 30 13:10:21.891723 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.891168    5837 scope.go:122] "RemoveContainer" containerID="01141dc5895af7f8abb7e47c9a0fdecd21088727db28a51324ede210d5af9281"
Oct 30 13:10:21.891723 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:21.891545    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01141dc5895af7f8abb7e47c9a0fdecd21088727db28a51324ede210d5af9281\": container with ID starting with 01141dc5895af7f8abb7e47c9a0fdecd21088727db28a51324ede210d5af9281 not found: ID does not exist" containerID="01141dc5895af7f8abb7e47c9a0fdecd21088727db28a51324ede210d5af9281"
Oct 30 13:10:21.891723 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.891584    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01141dc5895af7f8abb7e47c9a0fdecd21088727db28a51324ede210d5af9281"} err="failed to get container status \"01141dc5895af7f8abb7e47c9a0fdecd21088727db28a51324ede210d5af9281\": rpc error: code = NotFound desc = could not find container \"01141dc5895af7f8abb7e47c9a0fdecd21088727db28a51324ede210d5af9281\": container with ID starting with 01141dc5895af7f8abb7e47c9a0fdecd21088727db28a51324ede210d5af9281 not found: ID does not exist"
Oct 30 13:10:21.915635 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.915609    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"]
Oct 30 13:10:21.923380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.921883    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 13:10:21.923380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:21.922679    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7a590a/distinct-2-w1-7a590a-kserve-9d486856b-nx762"]
Oct 30 13:10:22.075628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.051751    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-tmp-dir" pod "089cb853-6545-4a2e-9aed-3b6cede912d6" (UID: "089cb853-6545-4a2e-9aed-3b6cede912d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:22.075628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.051474    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-tmp-dir\") pod \"089cb853-6545-4a2e-9aed-3b6cede912d6\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") "
Oct 30 13:10:22.075628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.051889    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-dshm\") pod \"089cb853-6545-4a2e-9aed-3b6cede912d6\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") "
Oct 30 13:10:22.075628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.052276    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-home\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-home\") pod \"089cb853-6545-4a2e-9aed-3b6cede912d6\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") "
Oct 30 13:10:22.075628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.052718    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-home" pod "089cb853-6545-4a2e-9aed-3b6cede912d6" (UID: "089cb853-6545-4a2e-9aed-3b6cede912d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:22.075628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.052784    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089cb853-6545-4a2e-9aed-3b6cede912d6-kube-api-access-dn467\" (UniqueName: \"kubernetes.io/projected/089cb853-6545-4a2e-9aed-3b6cede912d6-kube-api-access-dn467\") pod \"089cb853-6545-4a2e-9aed-3b6cede912d6\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") "
Oct 30 13:10:22.075628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.053149    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/089cb853-6545-4a2e-9aed-3b6cede912d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/089cb853-6545-4a2e-9aed-3b6cede912d6-tls-certs\") pod \"089cb853-6545-4a2e-9aed-3b6cede912d6\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") "
Oct 30 13:10:22.075628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.053588    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-model-cache\") pod \"089cb853-6545-4a2e-9aed-3b6cede912d6\" (UID: \"089cb853-6545-4a2e-9aed-3b6cede912d6\") "
Oct 30 13:10:22.075628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.053849    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-model-cache" pod "089cb853-6545-4a2e-9aed-3b6cede912d6" (UID: "089cb853-6545-4a2e-9aed-3b6cede912d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:22.075628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.054338    5837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-tmp-dir\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.075628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.054358    5837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-home\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.075628 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.054373    5837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-model-cache\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.076907 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.076851    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-dshm" pod "089cb853-6545-4a2e-9aed-3b6cede912d6" (UID: "089cb853-6545-4a2e-9aed-3b6cede912d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:22.096776 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.096729    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/089cb853-6545-4a2e-9aed-3b6cede912d6-tls-certs" pod "089cb853-6545-4a2e-9aed-3b6cede912d6" (UID: "089cb853-6545-4a2e-9aed-3b6cede912d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:22.105753 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.105560    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089cb853-6545-4a2e-9aed-3b6cede912d6-kube-api-access-dn467" pod "089cb853-6545-4a2e-9aed-3b6cede912d6" (UID: "089cb853-6545-4a2e-9aed-3b6cede912d6"). InnerVolumeSpecName "kube-api-access-dn467". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:22.163137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.163053    5837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089cb853-6545-4a2e-9aed-3b6cede912d6-dshm\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.163137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.163091    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn467\" (UniqueName: \"kubernetes.io/projected/089cb853-6545-4a2e-9aed-3b6cede912d6-kube-api-access-dn467\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.163137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.163107    5837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089cb853-6545-4a2e-9aed-3b6cede912d6-tls-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.893164 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.893122    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc" event={"ID":"089cb853-6545-4a2e-9aed-3b6cede912d6","Type":"ContainerDied","Data":"4229a79d5e8e814a1fd64d9fdcf28363421ca042e7de27d4e232fe9958e098bb"}
Oct 30 13:10:22.894322 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.894303    5837 scope.go:122] "RemoveContainer" containerID="4175e65a2184a1c266ff1f960db7f72fcd203e4b584fd92fc131624abe642770"
Oct 30 13:10:22.894801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:22.894779    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"
Oct 30 13:10:23.081316 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:23.079981    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"]
Oct 30 13:10:23.162473 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:23.158773    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/distinct-w4-53dcb1-kserve-779c4db7c-dltzc"]
Oct 30 13:10:23.654423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:23.653450    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" path="/var/lib/kubelet/pods/089cb853-6545-4a2e-9aed-3b6cede912d6/volumes"
Oct 30 13:10:23.654423 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:23.654128    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" path="/var/lib/kubelet/pods/d6eaa6ea-8274-49fa-b007-753de046e268/volumes"
Oct 30 13:10:23.851758 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:23.851696    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q" podUID="bc1998d0-b85e-46aa-a6d4-92b828b66d08" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:10:23.859645 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:23.859610    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9" podUID="ee36af90-164d-404c-be97-5dbaeafc0648" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:10:25.496342 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:25.495875    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:10:25.496342 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:25.495954    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:30.314136 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:30.314104    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"]
Oct 30 13:10:30.325206 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:30.325182    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a590a-wnp67"]
Oct 30 13:10:30.890707 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:30.890666    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv" podUID="3929deb3-fa21-4e2f-8ada-6348c9faac3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:10:30.890707 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:30.890697    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv" podUID="fb963087-5281-48f9-b322-110ae1590936" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:10:31.605684 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:31.605650    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ad506d-54ac-4607-a246-f7a36984de73" path="/var/lib/kubelet/pods/e3ad506d-54ac-4607-a246-f7a36984de73/volumes"
Oct 30 13:10:33.826342 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:33.826293    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9" podUID="ee36af90-164d-404c-be97-5dbaeafc0648" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:10:33.826783 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:33.826306    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q" podUID="bc1998d0-b85e-46aa-a6d4-92b828b66d08" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:10:36.926169 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:36.926144    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"
Oct 30 13:10:36.941016 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:36.940993    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"
Oct 30 13:10:37.034076 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.034040    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb963087-5281-48f9-b322-110ae1590936-kube-api-access-l982h\" (UniqueName: \"kubernetes.io/projected/fb963087-5281-48f9-b322-110ae1590936-kube-api-access-l982h\") pod \"fb963087-5281-48f9-b322-110ae1590936\" (UID: \"fb963087-5281-48f9-b322-110ae1590936\") "
Oct 30 13:10:37.034245 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.034132    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3929deb3-fa21-4e2f-8ada-6348c9faac3d-kube-api-access-2mnvk\" (UniqueName: \"kubernetes.io/projected/3929deb3-fa21-4e2f-8ada-6348c9faac3d-kube-api-access-2mnvk\") pod \"3929deb3-fa21-4e2f-8ada-6348c9faac3d\" (UID: \"3929deb3-fa21-4e2f-8ada-6348c9faac3d\") "
Oct 30 13:10:37.034296 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.034253    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3929deb3-fa21-4e2f-8ada-6348c9faac3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3929deb3-fa21-4e2f-8ada-6348c9faac3d-plugins-config-volume\") pod \"3929deb3-fa21-4e2f-8ada-6348c9faac3d\" (UID: \"3929deb3-fa21-4e2f-8ada-6348c9faac3d\") "
Oct 30 13:10:37.034348 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.034323    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb963087-5281-48f9-b322-110ae1590936-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb963087-5281-48f9-b322-110ae1590936-plugins-config-volume\") pod \"fb963087-5281-48f9-b322-110ae1590936\" (UID: \"fb963087-5281-48f9-b322-110ae1590936\") "
Oct 30 13:10:37.035073 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.034697    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3929deb3-fa21-4e2f-8ada-6348c9faac3d-plugins-config-volume" pod "3929deb3-fa21-4e2f-8ada-6348c9faac3d" (UID: "3929deb3-fa21-4e2f-8ada-6348c9faac3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:37.035073 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.034760    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb963087-5281-48f9-b322-110ae1590936-plugins-config-volume" pod "fb963087-5281-48f9-b322-110ae1590936" (UID: "fb963087-5281-48f9-b322-110ae1590936"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:37.035073 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.035041    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3929deb3-fa21-4e2f-8ada-6348c9faac3d-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:37.036529 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.036504    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb963087-5281-48f9-b322-110ae1590936-kube-api-access-l982h" pod "fb963087-5281-48f9-b322-110ae1590936" (UID: "fb963087-5281-48f9-b322-110ae1590936"). InnerVolumeSpecName "kube-api-access-l982h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:37.036757 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.036656    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3929deb3-fa21-4e2f-8ada-6348c9faac3d-kube-api-access-2mnvk" pod "3929deb3-fa21-4e2f-8ada-6348c9faac3d" (UID: "3929deb3-fa21-4e2f-8ada-6348c9faac3d"). InnerVolumeSpecName "kube-api-access-2mnvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:37.136179 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.136151    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l982h\" (UniqueName: \"kubernetes.io/projected/fb963087-5281-48f9-b322-110ae1590936-kube-api-access-l982h\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:37.136179 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.136179    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mnvk\" (UniqueName: \"kubernetes.io/projected/3929deb3-fa21-4e2f-8ada-6348c9faac3d-kube-api-access-2mnvk\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:37.136366 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.136189    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb963087-5281-48f9-b322-110ae1590936-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:37.193976 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.193900    5837 generic.go:356] "Generic (PLEG): container finished" podID="3929deb3-fa21-4e2f-8ada-6348c9faac3d" containerID="79ce72ddf662e4084f6df23a705dba8e1653abdcb6b252feb3d5dcf026fb666f" exitCode=0
Oct 30 13:10:37.193976 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.193945    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv" event={"ID":"3929deb3-fa21-4e2f-8ada-6348c9faac3d","Type":"ContainerDied","Data":"79ce72ddf662e4084f6df23a705dba8e1653abdcb6b252feb3d5dcf026fb666f"}
Oct 30 13:10:37.194216 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.194013    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv" event={"ID":"3929deb3-fa21-4e2f-8ada-6348c9faac3d","Type":"ContainerDied","Data":"453ba294961401d4d86d75858810a38e69af10b75a6b73085110a0a7d1f12033"}
Oct 30 13:10:37.194216 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.193957    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"
Oct 30 13:10:37.194216 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.194047    5837 scope.go:122] "RemoveContainer" containerID="79ce72ddf662e4084f6df23a705dba8e1653abdcb6b252feb3d5dcf026fb666f"
Oct 30 13:10:37.204609 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.196182    5837 generic.go:356] "Generic (PLEG): container finished" podID="fb963087-5281-48f9-b322-110ae1590936" containerID="c1fe7e5a94a10a38b594db3ef4fb013beadda9db2b247ca4757e96cb6d774ed6" exitCode=0
Oct 30 13:10:37.204609 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.196263    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"
Oct 30 13:10:37.204609 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.196331    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv" event={"ID":"fb963087-5281-48f9-b322-110ae1590936","Type":"ContainerDied","Data":"c1fe7e5a94a10a38b594db3ef4fb013beadda9db2b247ca4757e96cb6d774ed6"}
Oct 30 13:10:37.204609 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.196376    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv" event={"ID":"fb963087-5281-48f9-b322-110ae1590936","Type":"ContainerDied","Data":"52bc769bee5e546d257d8bd79af8fb3bee0c777a7e3ebb16f881889b7d744289"}
Oct 30 13:10:37.207438 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.207306    5837 scope.go:122] "RemoveContainer" containerID="79ce72ddf662e4084f6df23a705dba8e1653abdcb6b252feb3d5dcf026fb666f"
Oct 30 13:10:37.207726 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:37.207694    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ce72ddf662e4084f6df23a705dba8e1653abdcb6b252feb3d5dcf026fb666f\": container with ID starting with 79ce72ddf662e4084f6df23a705dba8e1653abdcb6b252feb3d5dcf026fb666f not found: ID does not exist" containerID="79ce72ddf662e4084f6df23a705dba8e1653abdcb6b252feb3d5dcf026fb666f"
Oct 30 13:10:37.207835 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.207724    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ce72ddf662e4084f6df23a705dba8e1653abdcb6b252feb3d5dcf026fb666f"} err="failed to get container status \"79ce72ddf662e4084f6df23a705dba8e1653abdcb6b252feb3d5dcf026fb666f\": rpc error: code = NotFound desc = could not find container \"79ce72ddf662e4084f6df23a705dba8e1653abdcb6b252feb3d5dcf026fb666f\": container with ID starting with 79ce72ddf662e4084f6df23a705dba8e1653abdcb6b252feb3d5dcf026fb666f not found: ID does not exist"
Oct 30 13:10:37.207835 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.207741    5837 scope.go:122] "RemoveContainer" containerID="c1fe7e5a94a10a38b594db3ef4fb013beadda9db2b247ca4757e96cb6d774ed6"
Oct 30 13:10:37.219010 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.218992    5837 scope.go:122] "RemoveContainer" containerID="c1fe7e5a94a10a38b594db3ef4fb013beadda9db2b247ca4757e96cb6d774ed6"
Oct 30 13:10:37.219302 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:37.219284    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1fe7e5a94a10a38b594db3ef4fb013beadda9db2b247ca4757e96cb6d774ed6\": container with ID starting with c1fe7e5a94a10a38b594db3ef4fb013beadda9db2b247ca4757e96cb6d774ed6 not found: ID does not exist" containerID="c1fe7e5a94a10a38b594db3ef4fb013beadda9db2b247ca4757e96cb6d774ed6"
Oct 30 13:10:37.219367 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.219310    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1fe7e5a94a10a38b594db3ef4fb013beadda9db2b247ca4757e96cb6d774ed6"} err="failed to get container status \"c1fe7e5a94a10a38b594db3ef4fb013beadda9db2b247ca4757e96cb6d774ed6\": rpc error: code = NotFound desc = could not find container \"c1fe7e5a94a10a38b594db3ef4fb013beadda9db2b247ca4757e96cb6d774ed6\": container with ID starting with c1fe7e5a94a10a38b594db3ef4fb013beadda9db2b247ca4757e96cb6d774ed6 not found: ID does not exist"
Oct 30 13:10:37.250922 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.250896    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"]
Oct 30 13:10:37.251118 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.251098    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" podUID="faf87439-5044-43cd-9b1a-200188299f24" containerName="istio-proxy" containerID="cri-o://45e9bf3f45778846fd02f494a46b8dfd062e6525e1c71dade9f7e3a24c1b3242" gracePeriod=30
Oct 30 13:10:37.260845 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.260821    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"]
Oct 30 13:10:37.261003 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.260983    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerName="istio-proxy" containerID="cri-o://3426311b73281d98ea110b5aea35377084bc5bd90ccc222b89c54d565cfadc54" gracePeriod=30
Oct 30 13:10:37.269837 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.269814    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"]
Oct 30 13:10:37.270013 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.269994    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" podUID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" containerName="istio-proxy" containerID="cri-o://853f2eab319a928f674fe993076a587a10a28af81f6c6221330b06e98f5d7748" gracePeriod=30
Oct 30 13:10:37.272467 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.272243    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"]
Oct 30 13:10:37.274545 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.274523    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"]
Oct 30 13:10:37.274696 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.274679    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy" containerID="cri-o://fa8ada98547aa343331b6af4d5932fb22fa08956d67800de17a7ba46412886a8" gracePeriod=30
Oct 30 13:10:37.280958 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.280925    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a590a-67fbb59666-9xvnv"]
Oct 30 13:10:37.313858 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.313832    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"]
Oct 30 13:10:37.317068 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.317042    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a590a-c79b8c955-f2xtv"]
Oct 30 13:10:37.503143 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.503113    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"]
Oct 30 13:10:37.503343 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.503312    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" podUID="25644850-37a6-4820-8ed2-4f00fedd8274" containerName="istio-proxy" containerID="cri-o://1356c126cff4b48144c5c1a1fcd668ab8c162254826981b5b3cbb2adaf014e58" gracePeriod=30
Oct 30 13:10:37.523796 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.523757    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"]
Oct 30 13:10:37.526945 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.526906    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk" podUID="f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" containerName="istio-proxy" containerID="cri-o://113bd1bf944aff569dad5b15129f28b20dde65581aafc9c5a41aafda5b1b53a1" gracePeriod=30
Oct 30 13:10:37.531076 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.531008    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"]
Oct 30 13:10:37.532025 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.531271    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4" podUID="f855b07d-e71c-491e-b430-0d33bdd77a5d" containerName="istio-proxy" containerID="cri-o://370162a2b3317705827ce0b480b824b505697369b008d014d1ada2def086e1e5" gracePeriod=30
Oct 30 13:10:37.533373 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.533351    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"]
Oct 30 13:10:37.534452 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.533619    5837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy" containerID="cri-o://37bd9503cb445a8d6d04c627d02af8b0cdbd18b892f0f9f4d83aeb74011f8f9b" gracePeriod=30
Oct 30 13:10:37.619812 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.619785    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3929deb3-fa21-4e2f-8ada-6348c9faac3d" path="/var/lib/kubelet/pods/3929deb3-fa21-4e2f-8ada-6348c9faac3d/volumes"
Oct 30 13:10:37.620444 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.620416    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb963087-5281-48f9-b322-110ae1590936" path="/var/lib/kubelet/pods/fb963087-5281-48f9-b322-110ae1590936/volumes"
Oct 30 13:10:37.670042 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.670020    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"
Oct 30 13:10:37.701311 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.701290    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"
Oct 30 13:10:37.743793 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.743769    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1998d0-b85e-46aa-a6d4-92b828b66d08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1998d0-b85e-46aa-a6d4-92b828b66d08-plugins-config-volume\") pod \"bc1998d0-b85e-46aa-a6d4-92b828b66d08\" (UID: \"bc1998d0-b85e-46aa-a6d4-92b828b66d08\") "
Oct 30 13:10:37.743964 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.743947    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1998d0-b85e-46aa-a6d4-92b828b66d08-kube-api-access-gw5dj\" (UniqueName: \"kubernetes.io/projected/bc1998d0-b85e-46aa-a6d4-92b828b66d08-kube-api-access-gw5dj\") pod \"bc1998d0-b85e-46aa-a6d4-92b828b66d08\" (UID: \"bc1998d0-b85e-46aa-a6d4-92b828b66d08\") "
Oct 30 13:10:37.744104 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.744084    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1998d0-b85e-46aa-a6d4-92b828b66d08-plugins-config-volume" pod "bc1998d0-b85e-46aa-a6d4-92b828b66d08" (UID: "bc1998d0-b85e-46aa-a6d4-92b828b66d08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:37.744362 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.744346    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1998d0-b85e-46aa-a6d4-92b828b66d08-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:37.746294 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.746267    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1998d0-b85e-46aa-a6d4-92b828b66d08-kube-api-access-gw5dj" pod "bc1998d0-b85e-46aa-a6d4-92b828b66d08" (UID: "bc1998d0-b85e-46aa-a6d4-92b828b66d08"). InnerVolumeSpecName "kube-api-access-gw5dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:37.845105 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.845081    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee36af90-164d-404c-be97-5dbaeafc0648-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee36af90-164d-404c-be97-5dbaeafc0648-plugins-config-volume\") pod \"ee36af90-164d-404c-be97-5dbaeafc0648\" (UID: \"ee36af90-164d-404c-be97-5dbaeafc0648\") "
Oct 30 13:10:37.845296 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.845132    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee36af90-164d-404c-be97-5dbaeafc0648-kube-api-access-jtv2b\" (UniqueName: \"kubernetes.io/projected/ee36af90-164d-404c-be97-5dbaeafc0648-kube-api-access-jtv2b\") pod \"ee36af90-164d-404c-be97-5dbaeafc0648\" (UID: \"ee36af90-164d-404c-be97-5dbaeafc0648\") "
Oct 30 13:10:37.845491 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.845467    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee36af90-164d-404c-be97-5dbaeafc0648-plugins-config-volume" pod "ee36af90-164d-404c-be97-5dbaeafc0648" (UID: "ee36af90-164d-404c-be97-5dbaeafc0648"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:37.845578 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.845521    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw5dj\" (UniqueName: \"kubernetes.io/projected/bc1998d0-b85e-46aa-a6d4-92b828b66d08-kube-api-access-gw5dj\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:37.847366 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.847343    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee36af90-164d-404c-be97-5dbaeafc0648-kube-api-access-jtv2b" pod "ee36af90-164d-404c-be97-5dbaeafc0648" (UID: "ee36af90-164d-404c-be97-5dbaeafc0648"). InnerVolumeSpecName "kube-api-access-jtv2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:37.946087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.946055    5837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee36af90-164d-404c-be97-5dbaeafc0648-plugins-config-volume\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:37.946087 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:37.946079    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtv2b\" (UniqueName: \"kubernetes.io/projected/ee36af90-164d-404c-be97-5dbaeafc0648-kube-api-access-jtv2b\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:38.203850 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.203770    5837 generic.go:356] "Generic (PLEG): container finished" podID="bc1998d0-b85e-46aa-a6d4-92b828b66d08" containerID="ac7b1dbb3c16a1bfe80cde0e1b6a64c91a979b62fb7df37c5173b036c2fee1db" exitCode=0
Oct 30 13:10:38.203850 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.203815    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"
Oct 30 13:10:38.204051 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.203858    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q" event={"ID":"bc1998d0-b85e-46aa-a6d4-92b828b66d08","Type":"ContainerDied","Data":"ac7b1dbb3c16a1bfe80cde0e1b6a64c91a979b62fb7df37c5173b036c2fee1db"}
Oct 30 13:10:38.204051 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.203911    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q" event={"ID":"bc1998d0-b85e-46aa-a6d4-92b828b66d08","Type":"ContainerDied","Data":"a6733e25188a844d147448cfeeb628127e136c6036346a00925503fb73c5f4d4"}
Oct 30 13:10:38.204051 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.203939    5837 scope.go:122] "RemoveContainer" containerID="ac7b1dbb3c16a1bfe80cde0e1b6a64c91a979b62fb7df37c5173b036c2fee1db"
Oct 30 13:10:38.207740 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.207713    5837 generic.go:356] "Generic (PLEG): container finished" podID="ee36af90-164d-404c-be97-5dbaeafc0648" containerID="155294b2dc20a7b61ec74a3be78fd3cdbd112919afc821e8ff008f839b3276b8" exitCode=0
Oct 30 13:10:38.207857 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.207754    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"
Oct 30 13:10:38.207857 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.207771    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9" event={"ID":"ee36af90-164d-404c-be97-5dbaeafc0648","Type":"ContainerDied","Data":"155294b2dc20a7b61ec74a3be78fd3cdbd112919afc821e8ff008f839b3276b8"}
Oct 30 13:10:38.207857 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.207809    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9" event={"ID":"ee36af90-164d-404c-be97-5dbaeafc0648","Type":"ContainerDied","Data":"27a430444a3b657f32dedff80b6123fc53523d34c4bae32b939cdf6b783a2ad5"}
Oct 30 13:10:38.216332 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.216312    5837 scope.go:122] "RemoveContainer" containerID="ac7b1dbb3c16a1bfe80cde0e1b6a64c91a979b62fb7df37c5173b036c2fee1db"
Oct 30 13:10:38.216654 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:38.216636    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac7b1dbb3c16a1bfe80cde0e1b6a64c91a979b62fb7df37c5173b036c2fee1db\": container with ID starting with ac7b1dbb3c16a1bfe80cde0e1b6a64c91a979b62fb7df37c5173b036c2fee1db not found: ID does not exist" containerID="ac7b1dbb3c16a1bfe80cde0e1b6a64c91a979b62fb7df37c5173b036c2fee1db"
Oct 30 13:10:38.216734 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.216660    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac7b1dbb3c16a1bfe80cde0e1b6a64c91a979b62fb7df37c5173b036c2fee1db"} err="failed to get container status \"ac7b1dbb3c16a1bfe80cde0e1b6a64c91a979b62fb7df37c5173b036c2fee1db\": rpc error: code = NotFound desc = could not find container \"ac7b1dbb3c16a1bfe80cde0e1b6a64c91a979b62fb7df37c5173b036c2fee1db\": container with ID starting with ac7b1dbb3c16a1bfe80cde0e1b6a64c91a979b62fb7df37c5173b036c2fee1db not found: ID does not exist"
Oct 30 13:10:38.216734 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.216677    5837 scope.go:122] "RemoveContainer" containerID="155294b2dc20a7b61ec74a3be78fd3cdbd112919afc821e8ff008f839b3276b8"
Oct 30 13:10:38.225772 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.225754    5837 scope.go:122] "RemoveContainer" containerID="155294b2dc20a7b61ec74a3be78fd3cdbd112919afc821e8ff008f839b3276b8"
Oct 30 13:10:38.226049 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:38.226030    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"155294b2dc20a7b61ec74a3be78fd3cdbd112919afc821e8ff008f839b3276b8\": container with ID starting with 155294b2dc20a7b61ec74a3be78fd3cdbd112919afc821e8ff008f839b3276b8 not found: ID does not exist" containerID="155294b2dc20a7b61ec74a3be78fd3cdbd112919afc821e8ff008f839b3276b8"
Oct 30 13:10:38.226114 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.226057    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"155294b2dc20a7b61ec74a3be78fd3cdbd112919afc821e8ff008f839b3276b8"} err="failed to get container status \"155294b2dc20a7b61ec74a3be78fd3cdbd112919afc821e8ff008f839b3276b8\": rpc error: code = NotFound desc = could not find container \"155294b2dc20a7b61ec74a3be78fd3cdbd112919afc821e8ff008f839b3276b8\": container with ID starting with 155294b2dc20a7b61ec74a3be78fd3cdbd112919afc821e8ff008f839b3276b8 not found: ID does not exist"
Oct 30 13:10:38.253067 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.253044    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"]
Oct 30 13:10:38.256896 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.256870    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-53dcb1-7c7b578b74-n765q"]
Oct 30 13:10:38.303135 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.303113    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"]
Oct 30 13:10:38.307175 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.307153    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-53dcb1-7f5b4bcbdc-zq6r9"]
Oct 30 13:10:38.528153 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:38.528122    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65c6d4fc99-x6wdt_4412a6ae-aaea-4aba-97f8-17ee44ff4dde/authorino/0.log"
Oct 30 13:10:39.369680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:39.369643    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-7bbc6bb9d7-tvp9w_6c9068e7-5aea-467f-ad19-b6421524c6de/kuadrant-console-plugin/0.log"
Oct 30 13:10:39.484834 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:39.484806    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:10:39.485013 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:39.484850    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:10:39.604440 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:39.604389    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1998d0-b85e-46aa-a6d4-92b828b66d08" path="/var/lib/kubelet/pods/bc1998d0-b85e-46aa-a6d4-92b828b66d08/volumes"
Oct 30 13:10:39.604801 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:39.604788    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee36af90-164d-404c-be97-5dbaeafc0648" path="/var/lib/kubelet/pods/ee36af90-164d-404c-be97-5dbaeafc0648/volumes"
Oct 30 13:10:39.682213 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:39.682139    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:10:39.682213 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:39.682189    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" podUID="faf87439-5044-43cd-9b1a-200188299f24" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:10:39.755357 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:39.755324    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:10:39.755553 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:39.755378    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:10:40.057829 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:40.057796    5837 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:10:40.058013 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:40.057847    5837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" podUID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:10:40.438196 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:40.438129    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-68669df9c4-2nng6_17d543a6-06a9-46ac-9b53-e43b0c0b34f5/manager/0.log"
Oct 30 13:10:42.778216 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.778152    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 13:10:42.783908 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.783883    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 13:10:42.787583 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.787564    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 13:10:42.792155 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.792135    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 13:10:42.796554 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.796535    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 13:10:42.876059 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.876030    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 13:10:42.883105 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.883081    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 13:10:42.886648 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.886627    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 13:10:42.896944 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.896913    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-workload-socket\") pod \"351759f4-8556-4412-952a-a9a63f93c23a\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") "
Oct 30 13:10:42.897060 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.896953    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25644850-37a6-4820-8ed2-4f00fedd8274-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25644850-37a6-4820-8ed2-4f00fedd8274-istio-podinfo\") pod \"25644850-37a6-4820-8ed2-4f00fedd8274\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") "
Oct 30 13:10:42.897060 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.896982    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25644850-37a6-4820-8ed2-4f00fedd8274-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25644850-37a6-4820-8ed2-4f00fedd8274-istiod-ca-cert\") pod \"25644850-37a6-4820-8ed2-4f00fedd8274\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") "
Oct 30 13:10:42.897060 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897009    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf87439-5044-43cd-9b1a-200188299f24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf87439-5044-43cd-9b1a-200188299f24-istiod-ca-cert\") pod \"faf87439-5044-43cd-9b1a-200188299f24\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") "
Oct 30 13:10:42.897060 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897036    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-workload-socket\") pod \"f24ec435-0ad6-4388-8a13-f38ca9a26800\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") "
Oct 30 13:10:42.897289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897064    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf87439-5044-43cd-9b1a-200188299f24-istio-token\" (UniqueName: \"kubernetes.io/projected/faf87439-5044-43cd-9b1a-200188299f24-istio-token\") pod \"faf87439-5044-43cd-9b1a-200188299f24\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") "
Oct 30 13:10:42.897289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897101    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faf87439-5044-43cd-9b1a-200188299f24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf87439-5044-43cd-9b1a-200188299f24-istio-podinfo\") pod \"faf87439-5044-43cd-9b1a-200188299f24\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") "
Oct 30 13:10:42.897289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897143    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-credential-socket\") pod \"25644850-37a6-4820-8ed2-4f00fedd8274\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") "
Oct 30 13:10:42.897289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897178    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24ec435-0ad6-4388-8a13-f38ca9a26800-kube-api-access-xznpt\" (UniqueName: \"kubernetes.io/projected/f24ec435-0ad6-4388-8a13-f38ca9a26800-kube-api-access-xznpt\") pod \"f24ec435-0ad6-4388-8a13-f38ca9a26800\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") "
Oct 30 13:10:42.897289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897233    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-workload-socket\") pod \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") "
Oct 30 13:10:42.897289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897221    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-workload-socket" pod "351759f4-8556-4412-952a-a9a63f93c23a" (UID: "351759f4-8556-4412-952a-a9a63f93c23a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.897289 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897266    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-token\" (UniqueName: \"kubernetes.io/projected/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-token\") pod \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") "
Oct 30 13:10:42.897680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897312    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-istio-envoy\") pod \"25644850-37a6-4820-8ed2-4f00fedd8274\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") "
Oct 30 13:10:42.897680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897347    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-workload-certs\") pod \"f24ec435-0ad6-4388-8a13-f38ca9a26800\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") "
Oct 30 13:10:42.897680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897378    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351759f4-8556-4412-952a-a9a63f93c23a-istio-token\" (UniqueName: \"kubernetes.io/projected/351759f4-8556-4412-952a-a9a63f93c23a-istio-token\") pod \"351759f4-8556-4412-952a-a9a63f93c23a\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") "
Oct 30 13:10:42.897680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897424    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/351759f4-8556-4412-952a-a9a63f93c23a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351759f4-8556-4412-952a-a9a63f93c23a-istio-podinfo\") pod \"351759f4-8556-4412-952a-a9a63f93c23a\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") "
Oct 30 13:10:42.897680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897474    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f24ec435-0ad6-4388-8a13-f38ca9a26800-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24ec435-0ad6-4388-8a13-f38ca9a26800-istiod-ca-cert\") pod \"f24ec435-0ad6-4388-8a13-f38ca9a26800\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") "
Oct 30 13:10:42.897680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897486    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf87439-5044-43cd-9b1a-200188299f24-istiod-ca-cert" pod "faf87439-5044-43cd-9b1a-200188299f24" (UID: "faf87439-5044-43cd-9b1a-200188299f24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:42.897680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897502    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf87439-5044-43cd-9b1a-200188299f24-kube-api-access-4jv49\" (UniqueName: \"kubernetes.io/projected/faf87439-5044-43cd-9b1a-200188299f24-kube-api-access-4jv49\") pod \"faf87439-5044-43cd-9b1a-200188299f24\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") "
Oct 30 13:10:42.897680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897522    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-credential-socket" pod "25644850-37a6-4820-8ed2-4f00fedd8274" (UID: "25644850-37a6-4820-8ed2-4f00fedd8274"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.897680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897552    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-credential-socket\") pod \"faf87439-5044-43cd-9b1a-200188299f24\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") "
Oct 30 13:10:42.897680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897589    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/351759f4-8556-4412-952a-a9a63f93c23a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351759f4-8556-4412-952a-a9a63f93c23a-istiod-ca-cert\") pod \"351759f4-8556-4412-952a-a9a63f93c23a\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") "
Oct 30 13:10:42.897680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897623    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-workload-certs\") pod \"25644850-37a6-4820-8ed2-4f00fedd8274\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") "
Oct 30 13:10:42.897680 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897654    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-podinfo\") pod \"f24ec435-0ad6-4388-8a13-f38ca9a26800\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897690    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-istio-envoy\") pod \"351759f4-8556-4412-952a-a9a63f93c23a\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897717    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-istio-data\") pod \"25644850-37a6-4820-8ed2-4f00fedd8274\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897745    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-credential-socket\") pod \"351759f4-8556-4412-952a-a9a63f93c23a\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897782    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-workload-socket\") pod \"faf87439-5044-43cd-9b1a-200188299f24\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897814    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-istio-data\") pod \"faf87439-5044-43cd-9b1a-200188299f24\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897850    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-workload-certs\") pod \"faf87439-5044-43cd-9b1a-200188299f24\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897901    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-data\") pod \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898001    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25644850-37a6-4820-8ed2-4f00fedd8274-istio-token\" (UniqueName: \"kubernetes.io/projected/25644850-37a6-4820-8ed2-4f00fedd8274-istio-token\") pod \"25644850-37a6-4820-8ed2-4f00fedd8274\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898030    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-credential-socket\") pod \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898064    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351759f4-8556-4412-952a-a9a63f93c23a-kube-api-access-hsxjl\" (UniqueName: \"kubernetes.io/projected/351759f4-8556-4412-952a-a9a63f93c23a-kube-api-access-hsxjl\") pod \"351759f4-8556-4412-952a-a9a63f93c23a\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898108    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-workload-certs\") pod \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898145    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-istio-data\") pod \"351759f4-8556-4412-952a-a9a63f93c23a\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898183    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-credential-socket\") pod \"f24ec435-0ad6-4388-8a13-f38ca9a26800\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898220    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-workload-certs\") pod \"351759f4-8556-4412-952a-a9a63f93c23a\" (UID: \"351759f4-8556-4412-952a-a9a63f93c23a\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898250    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ddf5f4-5772-45d0-8fbb-f1eba1931608-kube-api-access-jq5nw\" (UniqueName: \"kubernetes.io/projected/42ddf5f4-5772-45d0-8fbb-f1eba1931608-kube-api-access-jq5nw\") pod \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898289    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-token\" (UniqueName: \"kubernetes.io/projected/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-token\") pod \"f24ec435-0ad6-4388-8a13-f38ca9a26800\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898343    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-data\") pod \"f24ec435-0ad6-4388-8a13-f38ca9a26800\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898376    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25644850-37a6-4820-8ed2-4f00fedd8274-kube-api-access-rl89x\" (UniqueName: \"kubernetes.io/projected/25644850-37a6-4820-8ed2-4f00fedd8274-kube-api-access-rl89x\") pod \"25644850-37a6-4820-8ed2-4f00fedd8274\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898437    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istiod-ca-cert\") pod \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898466    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-podinfo\") pod \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") "
Oct 30 13:10:42.898498 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898495    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-istio-envoy\") pod \"faf87439-5044-43cd-9b1a-200188299f24\" (UID: \"faf87439-5044-43cd-9b1a-200188299f24\") "
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898527    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-workload-socket\") pod \"25644850-37a6-4820-8ed2-4f00fedd8274\" (UID: \"25644850-37a6-4820-8ed2-4f00fedd8274\") "
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898555    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-envoy\") pod \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\" (UID: \"42ddf5f4-5772-45d0-8fbb-f1eba1931608\") "
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898651    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-envoy\") pod \"f24ec435-0ad6-4388-8a13-f38ca9a26800\" (UID: \"f24ec435-0ad6-4388-8a13-f38ca9a26800\") "
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.900174    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.900196    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf87439-5044-43cd-9b1a-200188299f24-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.900210    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897585    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25644850-37a6-4820-8ed2-4f00fedd8274-istiod-ca-cert" pod "25644850-37a6-4820-8ed2-4f00fedd8274" (UID: "25644850-37a6-4820-8ed2-4f00fedd8274"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.897789    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-workload-socket" pod "f24ec435-0ad6-4388-8a13-f38ca9a26800" (UID: "f24ec435-0ad6-4388-8a13-f38ca9a26800"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898725    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-workload-certs" pod "42ddf5f4-5772-45d0-8fbb-f1eba1931608" (UID: "42ddf5f4-5772-45d0-8fbb-f1eba1931608"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.898964    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f24ec435-0ad6-4388-8a13-f38ca9a26800-istiod-ca-cert" pod "f24ec435-0ad6-4388-8a13-f38ca9a26800" (UID: "f24ec435-0ad6-4388-8a13-f38ca9a26800"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.899006    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-workload-certs" pod "f24ec435-0ad6-4388-8a13-f38ca9a26800" (UID: "f24ec435-0ad6-4388-8a13-f38ca9a26800"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.899280    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-workload-socket" pod "42ddf5f4-5772-45d0-8fbb-f1eba1931608" (UID: "42ddf5f4-5772-45d0-8fbb-f1eba1931608"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.899806    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-workload-certs" pod "faf87439-5044-43cd-9b1a-200188299f24" (UID: "faf87439-5044-43cd-9b1a-200188299f24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.899869    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-credential-socket" pod "faf87439-5044-43cd-9b1a-200188299f24" (UID: "faf87439-5044-43cd-9b1a-200188299f24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.900039    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-istio-data" pod "25644850-37a6-4820-8ed2-4f00fedd8274" (UID: "25644850-37a6-4820-8ed2-4f00fedd8274"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.900232    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-credential-socket" pod "351759f4-8556-4412-952a-a9a63f93c23a" (UID: "351759f4-8556-4412-952a-a9a63f93c23a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.900253    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351759f4-8556-4412-952a-a9a63f93c23a-istiod-ca-cert" pod "351759f4-8556-4412-952a-a9a63f93c23a" (UID: "351759f4-8556-4412-952a-a9a63f93c23a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.900468    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-data" pod "f24ec435-0ad6-4388-8a13-f38ca9a26800" (UID: "f24ec435-0ad6-4388-8a13-f38ca9a26800"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.900473    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-workload-socket" pod "faf87439-5044-43cd-9b1a-200188299f24" (UID: "faf87439-5044-43cd-9b1a-200188299f24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.900666    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-workload-certs" pod "25644850-37a6-4820-8ed2-4f00fedd8274" (UID: "25644850-37a6-4820-8ed2-4f00fedd8274"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.900690    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-istio-data" pod "351759f4-8556-4412-952a-a9a63f93c23a" (UID: "351759f4-8556-4412-952a-a9a63f93c23a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.900675    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-istio-data" pod "faf87439-5044-43cd-9b1a-200188299f24" (UID: "faf87439-5044-43cd-9b1a-200188299f24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.900879    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-credential-socket" pod "f24ec435-0ad6-4388-8a13-f38ca9a26800" (UID: "f24ec435-0ad6-4388-8a13-f38ca9a26800"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.901043    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-credential-socket" pod "42ddf5f4-5772-45d0-8fbb-f1eba1931608" (UID: "42ddf5f4-5772-45d0-8fbb-f1eba1931608"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.901137 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.901052    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-workload-certs" pod "351759f4-8556-4412-952a-a9a63f93c23a" (UID: "351759f4-8556-4412-952a-a9a63f93c23a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.903055 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.901267    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-workload-socket" pod "25644850-37a6-4820-8ed2-4f00fedd8274" (UID: "25644850-37a6-4820-8ed2-4f00fedd8274"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.903055 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.901805    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-data" pod "42ddf5f4-5772-45d0-8fbb-f1eba1931608" (UID: "42ddf5f4-5772-45d0-8fbb-f1eba1931608"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.904042 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.903914    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istiod-ca-cert" pod "42ddf5f4-5772-45d0-8fbb-f1eba1931608" (UID: "42ddf5f4-5772-45d0-8fbb-f1eba1931608"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:42.904042 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.904015    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ddf5f4-5772-45d0-8fbb-f1eba1931608-kube-api-access-jq5nw" pod "42ddf5f4-5772-45d0-8fbb-f1eba1931608" (UID: "42ddf5f4-5772-45d0-8fbb-f1eba1931608"). InnerVolumeSpecName "kube-api-access-jq5nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.904551 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.904502    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faf87439-5044-43cd-9b1a-200188299f24-istio-podinfo" pod "faf87439-5044-43cd-9b1a-200188299f24" (UID: "faf87439-5044-43cd-9b1a-200188299f24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:42.904551 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.904506    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf87439-5044-43cd-9b1a-200188299f24-kube-api-access-4jv49" pod "faf87439-5044-43cd-9b1a-200188299f24" (UID: "faf87439-5044-43cd-9b1a-200188299f24"). InnerVolumeSpecName "kube-api-access-4jv49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.905719 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.905589    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-istio-envoy" pod "faf87439-5044-43cd-9b1a-200188299f24" (UID: "faf87439-5044-43cd-9b1a-200188299f24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.905719 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.905636    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-envoy" pod "f24ec435-0ad6-4388-8a13-f38ca9a26800" (UID: "f24ec435-0ad6-4388-8a13-f38ca9a26800"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.905719 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.905648    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25644850-37a6-4820-8ed2-4f00fedd8274-istio-token" pod "25644850-37a6-4820-8ed2-4f00fedd8274" (UID: "25644850-37a6-4820-8ed2-4f00fedd8274"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.905719 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.905675    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-token" pod "42ddf5f4-5772-45d0-8fbb-f1eba1931608" (UID: "42ddf5f4-5772-45d0-8fbb-f1eba1931608"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.905719 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.905679    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24ec435-0ad6-4388-8a13-f38ca9a26800-kube-api-access-xznpt" pod "f24ec435-0ad6-4388-8a13-f38ca9a26800" (UID: "f24ec435-0ad6-4388-8a13-f38ca9a26800"). InnerVolumeSpecName "kube-api-access-xznpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.906763 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.906739    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/351759f4-8556-4412-952a-a9a63f93c23a-istio-podinfo" pod "351759f4-8556-4412-952a-a9a63f93c23a" (UID: "351759f4-8556-4412-952a-a9a63f93c23a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:42.907032 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.907001    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-token" pod "f24ec435-0ad6-4388-8a13-f38ca9a26800" (UID: "f24ec435-0ad6-4388-8a13-f38ca9a26800"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.907164 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.907113    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-podinfo" pod "f24ec435-0ad6-4388-8a13-f38ca9a26800" (UID: "f24ec435-0ad6-4388-8a13-f38ca9a26800"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:42.907242 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.907219    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25644850-37a6-4820-8ed2-4f00fedd8274-kube-api-access-rl89x" pod "25644850-37a6-4820-8ed2-4f00fedd8274" (UID: "25644850-37a6-4820-8ed2-4f00fedd8274"). InnerVolumeSpecName "kube-api-access-rl89x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.907315 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.907233    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351759f4-8556-4412-952a-a9a63f93c23a-istio-token" pod "351759f4-8556-4412-952a-a9a63f93c23a" (UID: "351759f4-8556-4412-952a-a9a63f93c23a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.907315 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.907240    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-istio-envoy" pod "25644850-37a6-4820-8ed2-4f00fedd8274" (UID: "25644850-37a6-4820-8ed2-4f00fedd8274"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.907315 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.907263    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25644850-37a6-4820-8ed2-4f00fedd8274-istio-podinfo" pod "25644850-37a6-4820-8ed2-4f00fedd8274" (UID: "25644850-37a6-4820-8ed2-4f00fedd8274"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:42.907568 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.907541    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf87439-5044-43cd-9b1a-200188299f24-istio-token" pod "faf87439-5044-43cd-9b1a-200188299f24" (UID: "faf87439-5044-43cd-9b1a-200188299f24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.907661 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.907563    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-envoy" pod "42ddf5f4-5772-45d0-8fbb-f1eba1931608" (UID: "42ddf5f4-5772-45d0-8fbb-f1eba1931608"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.907661 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.907648    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351759f4-8556-4412-952a-a9a63f93c23a-kube-api-access-hsxjl" pod "351759f4-8556-4412-952a-a9a63f93c23a" (UID: "351759f4-8556-4412-952a-a9a63f93c23a"). InnerVolumeSpecName "kube-api-access-hsxjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.908144 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.908120    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-podinfo" pod "42ddf5f4-5772-45d0-8fbb-f1eba1931608" (UID: "42ddf5f4-5772-45d0-8fbb-f1eba1931608"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:42.908780 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:42.908760    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-istio-envoy" pod "351759f4-8556-4412-952a-a9a63f93c23a" (UID: "351759f4-8556-4412-952a-a9a63f93c23a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.000918 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.000877    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-workload-socket\") pod \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") "
Oct 30 13:10:43.001119 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.000951    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-data\") pod \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") "
Oct 30 13:10:43.001119 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001003    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-token\" (UniqueName: \"kubernetes.io/projected/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-token\") pod \"f855b07d-e71c-491e-b430-0d33bdd77a5d\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") "
Oct 30 13:10:43.001119 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001038    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-workload-certs\") pod \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") "
Oct 30 13:10:43.001119 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001067    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-token\" (UniqueName: \"kubernetes.io/projected/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-token\") pod \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") "
Oct 30 13:10:43.001119 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001105    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-envoy\") pod \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") "
Oct 30 13:10:43.001380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001133    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-data\") pod \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") "
Oct 30 13:10:43.001380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001170    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istiod-ca-cert\") pod \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") "
Oct 30 13:10:43.001380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001205    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f855b07d-e71c-491e-b430-0d33bdd77a5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f855b07d-e71c-491e-b430-0d33bdd77a5d-istiod-ca-cert\") pod \"f855b07d-e71c-491e-b430-0d33bdd77a5d\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") "
Oct 30 13:10:43.001380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001254    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-podinfo\") pod \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") "
Oct 30 13:10:43.001380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001283    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-kube-api-access-8dtql\" (UniqueName: \"kubernetes.io/projected/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-kube-api-access-8dtql\") pod \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") "
Oct 30 13:10:43.001380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001312    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-workload-certs\") pod \"f855b07d-e71c-491e-b430-0d33bdd77a5d\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") "
Oct 30 13:10:43.001380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001362    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-envoy\") pod \"f855b07d-e71c-491e-b430-0d33bdd77a5d\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") "
Oct 30 13:10:43.001685 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001411    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istiod-ca-cert\") pod \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") "
Oct 30 13:10:43.001685 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001439    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-podinfo\") pod \"f855b07d-e71c-491e-b430-0d33bdd77a5d\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") "
Oct 30 13:10:43.001685 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001479    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-kube-api-access-jxzps\" (UniqueName: \"kubernetes.io/projected/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-kube-api-access-jxzps\") pod \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") "
Oct 30 13:10:43.001685 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001522    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-podinfo\") pod \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") "
Oct 30 13:10:43.001685 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001547    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-workload-socket\") pod \"f855b07d-e71c-491e-b430-0d33bdd77a5d\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") "
Oct 30 13:10:43.001685 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001583    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-credential-socket\") pod \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\" (UID: \"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd\") "
Oct 30 13:10:43.001685 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001611    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-token\" (UniqueName: \"kubernetes.io/projected/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-token\") pod \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") "
Oct 30 13:10:43.001685 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001641    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-envoy\") pod \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") "
Oct 30 13:10:43.001685 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001668    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-workload-socket\") pod \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") "
Oct 30 13:10:43.002123 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001693    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-workload-certs\") pod \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") "
Oct 30 13:10:43.002123 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001732    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-credential-socket\") pod \"f855b07d-e71c-491e-b430-0d33bdd77a5d\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") "
Oct 30 13:10:43.002123 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001751    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-data\") pod \"f855b07d-e71c-491e-b430-0d33bdd77a5d\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") "
Oct 30 13:10:43.002123 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001786    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-credential-socket\") pod \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\" (UID: \"80ea72c6-dacd-42c2-a7c1-c32ea4135e66\") "
Oct 30 13:10:43.002123 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.001823    5837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f855b07d-e71c-491e-b430-0d33bdd77a5d-kube-api-access-zfdjp\" (UniqueName: \"kubernetes.io/projected/f855b07d-e71c-491e-b430-0d33bdd77a5d-kube-api-access-zfdjp\") pod \"f855b07d-e71c-491e-b430-0d33bdd77a5d\" (UID: \"f855b07d-e71c-491e-b430-0d33bdd77a5d\") "
Oct 30 13:10:43.002359 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.002221    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xznpt\" (UniqueName: \"kubernetes.io/projected/f24ec435-0ad6-4388-8a13-f38ca9a26800-kube-api-access-xznpt\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.002359 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.002243    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.002359 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.002258    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.002359 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.002271    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.002359 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.002284    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.002359 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.002297    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351759f4-8556-4412-952a-a9a63f93c23a-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.002359 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.002309    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351759f4-8556-4412-952a-a9a63f93c23a-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.002359 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.002322    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24ec435-0ad6-4388-8a13-f38ca9a26800-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.002359 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.002336    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jv49\" (UniqueName: \"kubernetes.io/projected/faf87439-5044-43cd-9b1a-200188299f24-kube-api-access-4jv49\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.002359 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.002348    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.002359 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.002361    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351759f4-8556-4412-952a-a9a63f93c23a-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.002855 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.002375    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003285 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003256    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-workload-socket" pod "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" (UID: "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.003504 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003474    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-workload-socket" pod "f855b07d-e71c-491e-b430-0d33bdd77a5d" (UID: "f855b07d-e71c-491e-b430-0d33bdd77a5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.003608 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.002387    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003608 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003548    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003608 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003566    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003608 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003581    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003608 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003607    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003621    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003635    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003649    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003662    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25644850-37a6-4820-8ed2-4f00fedd8274-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003677    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003690    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsxjl\" (UniqueName: \"kubernetes.io/projected/351759f4-8556-4412-952a-a9a63f93c23a-kube-api-access-hsxjl\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003706    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003720    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003736    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003751    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351759f4-8556-4412-952a-a9a63f93c23a-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003767    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq5nw\" (UniqueName: \"kubernetes.io/projected/42ddf5f4-5772-45d0-8fbb-f1eba1931608-kube-api-access-jq5nw\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003780    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003794    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003806    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl89x\" (UniqueName: \"kubernetes.io/projected/25644850-37a6-4820-8ed2-4f00fedd8274-kube-api-access-rl89x\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003819    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003831    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003843    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf87439-5044-43cd-9b1a-200188299f24-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003858    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25644850-37a6-4820-8ed2-4f00fedd8274-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003870    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ddf5f4-5772-45d0-8fbb-f1eba1931608-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003883    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003896    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25644850-37a6-4820-8ed2-4f00fedd8274-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003908    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25644850-37a6-4820-8ed2-4f00fedd8274-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.003919 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003922    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24ec435-0ad6-4388-8a13-f38ca9a26800-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.005386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003936    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf87439-5044-43cd-9b1a-200188299f24-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.005386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003951    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf87439-5044-43cd-9b1a-200188299f24-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.005386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003498    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-data" pod "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" (UID: "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.005386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003716    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-credential-socket" pod "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" (UID: "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.005386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003785    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istiod-ca-cert" pod "80ea72c6-dacd-42c2-a7c1-c32ea4135e66" (UID: "80ea72c6-dacd-42c2-a7c1-c32ea4135e66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:43.005386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.003935    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-credential-socket" pod "f855b07d-e71c-491e-b430-0d33bdd77a5d" (UID: "f855b07d-e71c-491e-b430-0d33bdd77a5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.005386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.004191    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-data" pod "f855b07d-e71c-491e-b430-0d33bdd77a5d" (UID: "f855b07d-e71c-491e-b430-0d33bdd77a5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.005386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.004452    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-workload-socket" pod "80ea72c6-dacd-42c2-a7c1-c32ea4135e66" (UID: "80ea72c6-dacd-42c2-a7c1-c32ea4135e66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.005386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.004537    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-data" pod "80ea72c6-dacd-42c2-a7c1-c32ea4135e66" (UID: "80ea72c6-dacd-42c2-a7c1-c32ea4135e66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.005386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.004845    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-credential-socket" pod "80ea72c6-dacd-42c2-a7c1-c32ea4135e66" (UID: "80ea72c6-dacd-42c2-a7c1-c32ea4135e66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.005386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.004914    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istiod-ca-cert" pod "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" (UID: "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:43.005386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.005193    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-workload-certs" pod "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" (UID: "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.005386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.005238    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-workload-certs" pod "80ea72c6-dacd-42c2-a7c1-c32ea4135e66" (UID: "80ea72c6-dacd-42c2-a7c1-c32ea4135e66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.005386 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.005280    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f855b07d-e71c-491e-b430-0d33bdd77a5d-istiod-ca-cert" pod "f855b07d-e71c-491e-b430-0d33bdd77a5d" (UID: "f855b07d-e71c-491e-b430-0d33bdd77a5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:43.006167 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.005614    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-workload-certs" pod "f855b07d-e71c-491e-b430-0d33bdd77a5d" (UID: "f855b07d-e71c-491e-b430-0d33bdd77a5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.006707 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.006679    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-envoy" pod "80ea72c6-dacd-42c2-a7c1-c32ea4135e66" (UID: "80ea72c6-dacd-42c2-a7c1-c32ea4135e66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.007753 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.007730    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-podinfo" pod "80ea72c6-dacd-42c2-a7c1-c32ea4135e66" (UID: "80ea72c6-dacd-42c2-a7c1-c32ea4135e66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:43.008011 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.007988    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-token" pod "80ea72c6-dacd-42c2-a7c1-c32ea4135e66" (UID: "80ea72c6-dacd-42c2-a7c1-c32ea4135e66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:43.009457 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.009376    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-kube-api-access-8dtql" pod "80ea72c6-dacd-42c2-a7c1-c32ea4135e66" (UID: "80ea72c6-dacd-42c2-a7c1-c32ea4135e66"). InnerVolumeSpecName "kube-api-access-8dtql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:43.009457 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.009424    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f855b07d-e71c-491e-b430-0d33bdd77a5d-kube-api-access-zfdjp" pod "f855b07d-e71c-491e-b430-0d33bdd77a5d" (UID: "f855b07d-e71c-491e-b430-0d33bdd77a5d"). InnerVolumeSpecName "kube-api-access-zfdjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:43.009457 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.009386    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-podinfo" pod "f855b07d-e71c-491e-b430-0d33bdd77a5d" (UID: "f855b07d-e71c-491e-b430-0d33bdd77a5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:43.009632 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.009586    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-envoy" pod "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" (UID: "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.009825 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.009781    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-envoy" pod "f855b07d-e71c-491e-b430-0d33bdd77a5d" (UID: "f855b07d-e71c-491e-b430-0d33bdd77a5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:43.009825 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.009797    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-token" pod "f855b07d-e71c-491e-b430-0d33bdd77a5d" (UID: "f855b07d-e71c-491e-b430-0d33bdd77a5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:43.009825 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.009797    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-podinfo" pod "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" (UID: "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:43.009962 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.009822    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-kube-api-access-jxzps" pod "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" (UID: "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd"). InnerVolumeSpecName "kube-api-access-jxzps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:43.010744 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.010727    5837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-token" pod "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" (UID: "f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:43.104756 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104676    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.104756 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104706    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.104756 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104717    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxzps\" (UniqueName: \"kubernetes.io/projected/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-kube-api-access-jxzps\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.104756 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104727    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.104756 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104736    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.104756 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104746    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.104756 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104754    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104763    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104775    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104788    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104796    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104805    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104813    5837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-credential-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104820    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfdjp\" (UniqueName: \"kubernetes.io/projected/f855b07d-e71c-491e-b430-0d33bdd77a5d-kube-api-access-zfdjp\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104828    5837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-workload-socket\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104836    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104843    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104857    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104866    5837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-token\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104872    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104879    5837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-data\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104886    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104894    5837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f855b07d-e71c-491e-b430-0d33bdd77a5d-istiod-ca-cert\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104902    5837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-istio-podinfo\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104909    5837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dtql\" (UniqueName: \"kubernetes.io/projected/80ea72c6-dacd-42c2-a7c1-c32ea4135e66-kube-api-access-8dtql\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104918    5837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-workload-certs\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.105098 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.104928    5837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f855b07d-e71c-491e-b430-0d33bdd77a5d-istio-envoy\") on node \"ip-10-0-137-228.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.248846 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.248811    5837 generic.go:356] "Generic (PLEG): container finished" podID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" containerID="853f2eab319a928f674fe993076a587a10a28af81f6c6221330b06e98f5d7748" exitCode=0
Oct 30 13:10:43.249047 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.248875    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"
Oct 30 13:10:43.249047 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.248915    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" event={"ID":"42ddf5f4-5772-45d0-8fbb-f1eba1931608","Type":"ContainerDied","Data":"853f2eab319a928f674fe993076a587a10a28af81f6c6221330b06e98f5d7748"}
Oct 30 13:10:43.249047 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.249027    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x" event={"ID":"42ddf5f4-5772-45d0-8fbb-f1eba1931608","Type":"ContainerDied","Data":"5fdf4df2eb6b5b4d3ff14c8d7b1f34d091610056d29bf5239122295d21f33923"}
Oct 30 13:10:43.249199 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.249062    5837 scope.go:122] "RemoveContainer" containerID="853f2eab319a928f674fe993076a587a10a28af81f6c6221330b06e98f5d7748"
Oct 30 13:10:43.251458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.251255    5837 generic.go:356] "Generic (PLEG): container finished" podID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerID="3426311b73281d98ea110b5aea35377084bc5bd90ccc222b89c54d565cfadc54" exitCode=0
Oct 30 13:10:43.251458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.251326    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"
Oct 30 13:10:43.251458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.251346    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" event={"ID":"f24ec435-0ad6-4388-8a13-f38ca9a26800","Type":"ContainerDied","Data":"3426311b73281d98ea110b5aea35377084bc5bd90ccc222b89c54d565cfadc54"}
Oct 30 13:10:43.251458 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.251383    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x" event={"ID":"f24ec435-0ad6-4388-8a13-f38ca9a26800","Type":"ContainerDied","Data":"e17f32c562b47ee8d6517da48863aa5ad44ab9b4e4e8966e99b39bf38a19d428"}
Oct 30 13:10:43.260915 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.260888    5837 generic.go:356] "Generic (PLEG): container finished" podID="f855b07d-e71c-491e-b430-0d33bdd77a5d" containerID="370162a2b3317705827ce0b480b824b505697369b008d014d1ada2def086e1e5" exitCode=0
Oct 30 13:10:43.261133 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.260955    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"
Oct 30 13:10:43.261635 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.260972    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4" event={"ID":"f855b07d-e71c-491e-b430-0d33bdd77a5d","Type":"ContainerDied","Data":"370162a2b3317705827ce0b480b824b505697369b008d014d1ada2def086e1e5"}
Oct 30 13:10:43.261635 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.261598    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4" event={"ID":"f855b07d-e71c-491e-b430-0d33bdd77a5d","Type":"ContainerDied","Data":"01c698851232879aac3f55570099cbd649edf2e1afe9918a765e6f50c25c6b1d"}
Oct 30 13:10:43.263694 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.263672    5837 scope.go:122] "RemoveContainer" containerID="853f2eab319a928f674fe993076a587a10a28af81f6c6221330b06e98f5d7748"
Oct 30 13:10:43.264302 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:43.264190    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853f2eab319a928f674fe993076a587a10a28af81f6c6221330b06e98f5d7748\": container with ID starting with 853f2eab319a928f674fe993076a587a10a28af81f6c6221330b06e98f5d7748 not found: ID does not exist" containerID="853f2eab319a928f674fe993076a587a10a28af81f6c6221330b06e98f5d7748"
Oct 30 13:10:43.264302 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.264224    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853f2eab319a928f674fe993076a587a10a28af81f6c6221330b06e98f5d7748"} err="failed to get container status \"853f2eab319a928f674fe993076a587a10a28af81f6c6221330b06e98f5d7748\": rpc error: code = NotFound desc = could not find container \"853f2eab319a928f674fe993076a587a10a28af81f6c6221330b06e98f5d7748\": container with ID starting with 853f2eab319a928f674fe993076a587a10a28af81f6c6221330b06e98f5d7748 not found: ID does not exist"
Oct 30 13:10:43.264302 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.264246    5837 scope.go:122] "RemoveContainer" containerID="3426311b73281d98ea110b5aea35377084bc5bd90ccc222b89c54d565cfadc54"
Oct 30 13:10:43.266174 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.266150    5837 generic.go:356] "Generic (PLEG): container finished" podID="faf87439-5044-43cd-9b1a-200188299f24" containerID="45e9bf3f45778846fd02f494a46b8dfd062e6525e1c71dade9f7e3a24c1b3242" exitCode=0
Oct 30 13:10:43.266271 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.266253    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" event={"ID":"faf87439-5044-43cd-9b1a-200188299f24","Type":"ContainerDied","Data":"45e9bf3f45778846fd02f494a46b8dfd062e6525e1c71dade9f7e3a24c1b3242"}
Oct 30 13:10:43.266314 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.266296    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm" event={"ID":"faf87439-5044-43cd-9b1a-200188299f24","Type":"ContainerDied","Data":"3d7655000cbba8d1ad2f0f6202a60c14a731c641d71b91258fe9ab78ead0c175"}
Oct 30 13:10:43.266380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.266366    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"
Oct 30 13:10:43.268961 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.268940    5837 generic.go:356] "Generic (PLEG): container finished" podID="f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" containerID="113bd1bf944aff569dad5b15129f28b20dde65581aafc9c5a41aafda5b1b53a1" exitCode=0
Oct 30 13:10:43.269063 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.269022    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk" event={"ID":"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd","Type":"ContainerDied","Data":"113bd1bf944aff569dad5b15129f28b20dde65581aafc9c5a41aafda5b1b53a1"}
Oct 30 13:10:43.269063 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.269060    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk" event={"ID":"f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd","Type":"ContainerDied","Data":"6b3478d8ac67a0fbae0d0f95426a453f7e6237ff481b0e276cb825015e41aff0"}
Oct 30 13:10:43.269194 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.269125    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"
Oct 30 13:10:43.272094 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.272070    5837 generic.go:356] "Generic (PLEG): container finished" podID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerID="37bd9503cb445a8d6d04c627d02af8b0cdbd18b892f0f9f4d83aeb74011f8f9b" exitCode=0
Oct 30 13:10:43.272199 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.272151    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" event={"ID":"80ea72c6-dacd-42c2-a7c1-c32ea4135e66","Type":"ContainerDied","Data":"37bd9503cb445a8d6d04c627d02af8b0cdbd18b892f0f9f4d83aeb74011f8f9b"}
Oct 30 13:10:43.272371 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.272190    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q" event={"ID":"80ea72c6-dacd-42c2-a7c1-c32ea4135e66","Type":"ContainerDied","Data":"a67dffe96879eb386c71bee5d7e09fe06dc45faf35898c6db76a9c5a4b164c30"}
Oct 30 13:10:43.272371 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.272278    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"
Oct 30 13:10:43.275083 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.275061    5837 generic.go:356] "Generic (PLEG): container finished" podID="25644850-37a6-4820-8ed2-4f00fedd8274" containerID="1356c126cff4b48144c5c1a1fcd668ab8c162254826981b5b3cbb2adaf014e58" exitCode=0
Oct 30 13:10:43.275233 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.275217    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" event={"ID":"25644850-37a6-4820-8ed2-4f00fedd8274","Type":"ContainerDied","Data":"1356c126cff4b48144c5c1a1fcd668ab8c162254826981b5b3cbb2adaf014e58"}
Oct 30 13:10:43.275304 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.275260    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2" event={"ID":"25644850-37a6-4820-8ed2-4f00fedd8274","Type":"ContainerDied","Data":"530f85b8be1110cd90cca74f0f1e7762b9b92f1f964f6ced9b386430d40c6ef5"}
Oct 30 13:10:43.275371 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.275342    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"
Oct 30 13:10:43.278859 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.278840    5837 scope.go:122] "RemoveContainer" containerID="3426311b73281d98ea110b5aea35377084bc5bd90ccc222b89c54d565cfadc54"
Oct 30 13:10:43.279202 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:43.279173    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3426311b73281d98ea110b5aea35377084bc5bd90ccc222b89c54d565cfadc54\": container with ID starting with 3426311b73281d98ea110b5aea35377084bc5bd90ccc222b89c54d565cfadc54 not found: ID does not exist" containerID="3426311b73281d98ea110b5aea35377084bc5bd90ccc222b89c54d565cfadc54"
Oct 30 13:10:43.279298 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.279209    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3426311b73281d98ea110b5aea35377084bc5bd90ccc222b89c54d565cfadc54"} err="failed to get container status \"3426311b73281d98ea110b5aea35377084bc5bd90ccc222b89c54d565cfadc54\": rpc error: code = NotFound desc = could not find container \"3426311b73281d98ea110b5aea35377084bc5bd90ccc222b89c54d565cfadc54\": container with ID starting with 3426311b73281d98ea110b5aea35377084bc5bd90ccc222b89c54d565cfadc54 not found: ID does not exist"
Oct 30 13:10:43.279298 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.279236    5837 scope.go:122] "RemoveContainer" containerID="370162a2b3317705827ce0b480b824b505697369b008d014d1ada2def086e1e5"
Oct 30 13:10:43.280744 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.280721    5837 generic.go:356] "Generic (PLEG): container finished" podID="351759f4-8556-4412-952a-a9a63f93c23a" containerID="fa8ada98547aa343331b6af4d5932fb22fa08956d67800de17a7ba46412886a8" exitCode=0
Oct 30 13:10:43.280844 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.280797    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" event={"ID":"351759f4-8556-4412-952a-a9a63f93c23a","Type":"ContainerDied","Data":"fa8ada98547aa343331b6af4d5932fb22fa08956d67800de17a7ba46412886a8"}
Oct 30 13:10:43.280844 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.280834    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh" event={"ID":"351759f4-8556-4412-952a-a9a63f93c23a","Type":"ContainerDied","Data":"09e4468a2b182a705b652561b83fb1a67f1b175eebbf070bf4b16d21c57d8fd2"}
Oct 30 13:10:43.281123 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.280899    5837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"
Oct 30 13:10:43.330594 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.330553    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"]
Oct 30 13:10:43.331961 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.331885    5837 scope.go:122] "RemoveContainer" containerID="370162a2b3317705827ce0b480b824b505697369b008d014d1ada2def086e1e5"
Oct 30 13:10:43.332258 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:43.332233    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370162a2b3317705827ce0b480b824b505697369b008d014d1ada2def086e1e5\": container with ID starting with 370162a2b3317705827ce0b480b824b505697369b008d014d1ada2def086e1e5 not found: ID does not exist" containerID="370162a2b3317705827ce0b480b824b505697369b008d014d1ada2def086e1e5"
Oct 30 13:10:43.332380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.332271    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370162a2b3317705827ce0b480b824b505697369b008d014d1ada2def086e1e5"} err="failed to get container status \"370162a2b3317705827ce0b480b824b505697369b008d014d1ada2def086e1e5\": rpc error: code = NotFound desc = could not find container \"370162a2b3317705827ce0b480b824b505697369b008d014d1ada2def086e1e5\": container with ID starting with 370162a2b3317705827ce0b480b824b505697369b008d014d1ada2def086e1e5 not found: ID does not exist"
Oct 30 13:10:43.332380 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.332295    5837 scope.go:122] "RemoveContainer" containerID="45e9bf3f45778846fd02f494a46b8dfd062e6525e1c71dade9f7e3a24c1b3242"
Oct 30 13:10:43.335901 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.335879    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-lkv7x"]
Oct 30 13:10:43.344616 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.344599    5837 scope.go:122] "RemoveContainer" containerID="45e9bf3f45778846fd02f494a46b8dfd062e6525e1c71dade9f7e3a24c1b3242"
Oct 30 13:10:43.344902 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:43.344887    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e9bf3f45778846fd02f494a46b8dfd062e6525e1c71dade9f7e3a24c1b3242\": container with ID starting with 45e9bf3f45778846fd02f494a46b8dfd062e6525e1c71dade9f7e3a24c1b3242 not found: ID does not exist" containerID="45e9bf3f45778846fd02f494a46b8dfd062e6525e1c71dade9f7e3a24c1b3242"
Oct 30 13:10:43.344970 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.344908    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e9bf3f45778846fd02f494a46b8dfd062e6525e1c71dade9f7e3a24c1b3242"} err="failed to get container status \"45e9bf3f45778846fd02f494a46b8dfd062e6525e1c71dade9f7e3a24c1b3242\": rpc error: code = NotFound desc = could not find container \"45e9bf3f45778846fd02f494a46b8dfd062e6525e1c71dade9f7e3a24c1b3242\": container with ID starting with 45e9bf3f45778846fd02f494a46b8dfd062e6525e1c71dade9f7e3a24c1b3242 not found: ID does not exist"
Oct 30 13:10:43.344970 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.344924    5837 scope.go:122] "RemoveContainer" containerID="113bd1bf944aff569dad5b15129f28b20dde65581aafc9c5a41aafda5b1b53a1"
Oct 30 13:10:43.354023 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.354004    5837 scope.go:122] "RemoveContainer" containerID="113bd1bf944aff569dad5b15129f28b20dde65581aafc9c5a41aafda5b1b53a1"
Oct 30 13:10:43.354298 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:43.354279    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113bd1bf944aff569dad5b15129f28b20dde65581aafc9c5a41aafda5b1b53a1\": container with ID starting with 113bd1bf944aff569dad5b15129f28b20dde65581aafc9c5a41aafda5b1b53a1 not found: ID does not exist" containerID="113bd1bf944aff569dad5b15129f28b20dde65581aafc9c5a41aafda5b1b53a1"
Oct 30 13:10:43.354345 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.354305    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113bd1bf944aff569dad5b15129f28b20dde65581aafc9c5a41aafda5b1b53a1"} err="failed to get container status \"113bd1bf944aff569dad5b15129f28b20dde65581aafc9c5a41aafda5b1b53a1\": rpc error: code = NotFound desc = could not find container \"113bd1bf944aff569dad5b15129f28b20dde65581aafc9c5a41aafda5b1b53a1\": container with ID starting with 113bd1bf944aff569dad5b15129f28b20dde65581aafc9c5a41aafda5b1b53a1 not found: ID does not exist"
Oct 30 13:10:43.354345 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.354321    5837 scope.go:122] "RemoveContainer" containerID="37bd9503cb445a8d6d04c627d02af8b0cdbd18b892f0f9f4d83aeb74011f8f9b"
Oct 30 13:10:43.382613 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.379603    5837 scope.go:122] "RemoveContainer" containerID="37bd9503cb445a8d6d04c627d02af8b0cdbd18b892f0f9f4d83aeb74011f8f9b"
Oct 30 13:10:43.382613 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:43.380016    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37bd9503cb445a8d6d04c627d02af8b0cdbd18b892f0f9f4d83aeb74011f8f9b\": container with ID starting with 37bd9503cb445a8d6d04c627d02af8b0cdbd18b892f0f9f4d83aeb74011f8f9b not found: ID does not exist" containerID="37bd9503cb445a8d6d04c627d02af8b0cdbd18b892f0f9f4d83aeb74011f8f9b"
Oct 30 13:10:43.382613 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.380058    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bd9503cb445a8d6d04c627d02af8b0cdbd18b892f0f9f4d83aeb74011f8f9b"} err="failed to get container status \"37bd9503cb445a8d6d04c627d02af8b0cdbd18b892f0f9f4d83aeb74011f8f9b\": rpc error: code = NotFound desc = could not find container \"37bd9503cb445a8d6d04c627d02af8b0cdbd18b892f0f9f4d83aeb74011f8f9b\": container with ID starting with 37bd9503cb445a8d6d04c627d02af8b0cdbd18b892f0f9f4d83aeb74011f8f9b not found: ID does not exist"
Oct 30 13:10:43.382613 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.380077    5837 scope.go:122] "RemoveContainer" containerID="1356c126cff4b48144c5c1a1fcd668ab8c162254826981b5b3cbb2adaf014e58"
Oct 30 13:10:43.392518 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.392500    5837 scope.go:122] "RemoveContainer" containerID="23974a71b9810c8bd9b28fb891a606cff350e664ba3fda6e41453f8ed2a22c8a"
Oct 30 13:10:43.405439 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.405363    5837 scope.go:122] "RemoveContainer" containerID="1356c126cff4b48144c5c1a1fcd668ab8c162254826981b5b3cbb2adaf014e58"
Oct 30 13:10:43.405807 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:43.405753    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1356c126cff4b48144c5c1a1fcd668ab8c162254826981b5b3cbb2adaf014e58\": container with ID starting with 1356c126cff4b48144c5c1a1fcd668ab8c162254826981b5b3cbb2adaf014e58 not found: ID does not exist" containerID="1356c126cff4b48144c5c1a1fcd668ab8c162254826981b5b3cbb2adaf014e58"
Oct 30 13:10:43.405807 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.405790    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1356c126cff4b48144c5c1a1fcd668ab8c162254826981b5b3cbb2adaf014e58"} err="failed to get container status \"1356c126cff4b48144c5c1a1fcd668ab8c162254826981b5b3cbb2adaf014e58\": rpc error: code = NotFound desc = could not find container \"1356c126cff4b48144c5c1a1fcd668ab8c162254826981b5b3cbb2adaf014e58\": container with ID starting with 1356c126cff4b48144c5c1a1fcd668ab8c162254826981b5b3cbb2adaf014e58 not found: ID does not exist"
Oct 30 13:10:43.405807 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.405813    5837 scope.go:122] "RemoveContainer" containerID="23974a71b9810c8bd9b28fb891a606cff350e664ba3fda6e41453f8ed2a22c8a"
Oct 30 13:10:43.406191 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:43.406162    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23974a71b9810c8bd9b28fb891a606cff350e664ba3fda6e41453f8ed2a22c8a\": container with ID starting with 23974a71b9810c8bd9b28fb891a606cff350e664ba3fda6e41453f8ed2a22c8a not found: ID does not exist" containerID="23974a71b9810c8bd9b28fb891a606cff350e664ba3fda6e41453f8ed2a22c8a"
Oct 30 13:10:43.406245 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.406206    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23974a71b9810c8bd9b28fb891a606cff350e664ba3fda6e41453f8ed2a22c8a"} err="failed to get container status \"23974a71b9810c8bd9b28fb891a606cff350e664ba3fda6e41453f8ed2a22c8a\": rpc error: code = NotFound desc = could not find container \"23974a71b9810c8bd9b28fb891a606cff350e664ba3fda6e41453f8ed2a22c8a\": container with ID starting with 23974a71b9810c8bd9b28fb891a606cff350e664ba3fda6e41453f8ed2a22c8a not found: ID does not exist"
Oct 30 13:10:43.406245 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.406231    5837 scope.go:122] "RemoveContainer" containerID="fa8ada98547aa343331b6af4d5932fb22fa08956d67800de17a7ba46412886a8"
Oct 30 13:10:43.423081 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.423050    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"]
Oct 30 13:10:43.423879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.423854    5837 scope.go:122] "RemoveContainer" containerID="fa8ada98547aa343331b6af4d5932fb22fa08956d67800de17a7ba46412886a8"
Oct 30 13:10:43.425961 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:43.425935    5837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8ada98547aa343331b6af4d5932fb22fa08956d67800de17a7ba46412886a8\": container with ID starting with fa8ada98547aa343331b6af4d5932fb22fa08956d67800de17a7ba46412886a8 not found: ID does not exist" containerID="fa8ada98547aa343331b6af4d5932fb22fa08956d67800de17a7ba46412886a8"
Oct 30 13:10:43.426071 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.425967    5837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8ada98547aa343331b6af4d5932fb22fa08956d67800de17a7ba46412886a8"} err="failed to get container status \"fa8ada98547aa343331b6af4d5932fb22fa08956d67800de17a7ba46412886a8\": rpc error: code = NotFound desc = could not find container \"fa8ada98547aa343331b6af4d5932fb22fa08956d67800de17a7ba46412886a8\": container with ID starting with fa8ada98547aa343331b6af4d5932fb22fa08956d67800de17a7ba46412886a8 not found: ID does not exist"
Oct 30 13:10:43.427673 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.427649    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tskq4"]
Oct 30 13:10:43.460112 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.460085    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"]
Oct 30 13:10:43.466204 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.466177    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-thm8x"]
Oct 30 13:10:43.538222 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.538169    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"]
Oct 30 13:10:43.542826 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.542799    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-tvjm2"]
Oct 30 13:10:43.575989 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.575960    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"]
Oct 30 13:10:43.582581 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.582536    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-vh6vh"]
Oct 30 13:10:43.605117 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.605089    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25644850-37a6-4820-8ed2-4f00fedd8274" path="/var/lib/kubelet/pods/25644850-37a6-4820-8ed2-4f00fedd8274/volumes"
Oct 30 13:10:43.605580 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.605538    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="351759f4-8556-4412-952a-a9a63f93c23a" path="/var/lib/kubelet/pods/351759f4-8556-4412-952a-a9a63f93c23a/volumes"
Oct 30 13:10:43.605923 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.605911    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" path="/var/lib/kubelet/pods/42ddf5f4-5772-45d0-8fbb-f1eba1931608/volumes"
Oct 30 13:10:43.606618 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.606604    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" path="/var/lib/kubelet/pods/f24ec435-0ad6-4388-8a13-f38ca9a26800/volumes"
Oct 30 13:10:43.606978 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.606966    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f855b07d-e71c-491e-b430-0d33bdd77a5d" path="/var/lib/kubelet/pods/f855b07d-e71c-491e-b430-0d33bdd77a5d/volumes"
Oct 30 13:10:43.615327 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.615307    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"]
Oct 30 13:10:43.625307 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.624312    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xlmxk"]
Oct 30 13:10:43.656559 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.656528    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"]
Oct 30 13:10:43.662231 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.662187    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-kbjvm"]
Oct 30 13:10:43.695147 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.695112    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"]
Oct 30 13:10:43.701047 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:43.701023    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-qbq5q"]
Oct 30 13:10:44.100717 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:44.100690    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65c6d4fc99-x6wdt_4412a6ae-aaea-4aba-97f8-17ee44ff4dde/authorino/0.log"
Oct 30 13:10:45.092699 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:10:45.092665    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:10:45.604061 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:45.604029    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" path="/var/lib/kubelet/pods/80ea72c6-dacd-42c2-a7c1-c32ea4135e66/volumes"
Oct 30 13:10:45.604541 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:45.604526    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" path="/var/lib/kubelet/pods/f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd/volumes"
Oct 30 13:10:45.604995 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:45.604981    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faf87439-5044-43cd-9b1a-200188299f24" path="/var/lib/kubelet/pods/faf87439-5044-43cd-9b1a-200188299f24/volumes"
Oct 30 13:10:57.588132 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:57.588101    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1f06d9ee-vz6t5_cb5cc095-af95-4463-9e95-000901d19a91/revoke-keys/0.log"
Oct 30 13:10:57.768804 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:57.768779    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-300b7a2e-hg845_1dabef2b-77b1-4a05-8c5c-c12e3e3d03a5/revoke-keys/0.log"
Oct 30 13:10:58.118624 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:58.118597    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-41ce7966-jd7mq_7b0047c7-c493-412a-83e1-10b48bdc06b6/revoke-keys/0.log"
Oct 30 13:10:58.185657 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:58.185632    5837 scope.go:122] "RemoveContainer" containerID="bc133416c9a9331b05457c01458d0ed8bc908e876db3cfb55f96e3d2b4b66631"
Oct 30 13:10:58.195326 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:58.195306    5837 scope.go:122] "RemoveContainer" containerID="1767a306031a69eada49e1b2d4bad043f292d904c2c948602bc8c41a93b2f892"
Oct 30 13:10:58.203952 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:58.203935    5837 scope.go:122] "RemoveContainer" containerID="6a48b9a09db21bb004cfea570a57563173e3a03512df3910f2d31e3ad54a8da3"
Oct 30 13:10:58.302909 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:58.302883    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-54e7adec-jwpqb_1cf688ee-b349-40af-a2cc-14877d335e34/revoke-keys/0.log"
Oct 30 13:10:58.489911 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:58.489885    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5b2fb3a2-g8c2p_4348ac37-952a-4000-ba42-3a8fa6f12c4d/revoke-keys/0.log"
Oct 30 13:10:58.680743 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:58.680719    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-727805cf-zbqr9_6e75167c-1f70-44d3-b782-f6c7c4ff64d1/revoke-keys/0.log"
Oct 30 13:10:58.848545 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:58.848479    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7d9a8b3f-5ncq8_15f36d2f-d742-4b02-a0f6-2ea63c70f471/revoke-keys/0.log"
Oct 30 13:10:59.013813 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:59.013788    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9699cd68-6rl6v_f3cb822c-3e4d-4db6-86e0-ca099e164daf/revoke-keys/0.log"
Oct 30 13:10:59.203501 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:59.203434    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9e32c90f-hzj97_5c6879e6-651c-4a62-b0fa-511232557e12/revoke-keys/0.log"
Oct 30 13:10:59.391992 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:10:59.391968    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ad83ceac-hljf7_b331e440-2ecf-4b68-ad58-d593926ad417/revoke-keys/0.log"
Oct 30 13:11:01.032761 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:01.032730    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65c6d4fc99-x6wdt_4412a6ae-aaea-4aba-97f8-17ee44ff4dde/authorino/0.log"
Oct 30 13:11:01.776671 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:01.776646    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-7bbc6bb9d7-tvp9w_6c9068e7-5aea-467f-ad19-b6421524c6de/kuadrant-console-plugin/0.log"
Oct 30 13:11:02.311048 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:02.311021    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-68669df9c4-2nng6_17d543a6-06a9-46ac-9b53-e43b0c0b34f5/manager/0.log"
Oct 30 13:11:03.815155 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:03.815123    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kvdnb_c5613511-ebe4-44bb-9d1b-a0652e3f97b8/discovery/0.log"
Oct 30 13:11:04.197272 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:04.197205    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-dc45784b6-t2t85_5d0188ec-41eb-45b4-ae5c-705341675edf/kube-auth-proxy/0.log"
Oct 30 13:11:04.757781 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:04.757747    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-xhs8n_99edc7cd-e5d4-4ac9-ab07-c0b91926a654/istio-proxy/0.log"
Oct 30 13:11:06.028897 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:06.028864    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"]
Oct 30 13:11:06.040329 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:06.040300    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f06d9ee-vz6t5"]
Oct 30 13:11:06.459377 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:06.459306    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-pdnxs_20d9a8aa-0955-49d8-bb05-d1a6fa52f560/main/0.log"
Oct 30 13:11:06.810311 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:06.810284    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-vhr7q_c06bb10a-6170-4c87-82e8-4695cdddc789/main/0.log"
Oct 30 13:11:07.147970 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:07.147899    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8wtq4_80f38956-a205-4c3a-ab46-aa50d995cc2d/main/0.log"
Oct 30 13:11:07.338223 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:07.338197    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-sj8tq_b175788a-b7a3-4aab-a17f-1362ac6b83b5/main/0.log"
Oct 30 13:11:07.603279 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:07.603249    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb5cc095-af95-4463-9e95-000901d19a91" path="/var/lib/kubelet/pods/cb5cc095-af95-4463-9e95-000901d19a91/volumes"
Oct 30 13:11:15.093909 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:11:15.093868    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:11:24.136865 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:24.136836    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kd88h_0b894531-bbb4-4e28-b4d4-ac51d5ed4ca4/global-pull-secret-syncer/0.log"
Oct 30 13:11:24.930364 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:24.930333    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n6d9z_bbb54d63-c3cc-4759-ba36-252e1ffc4d18/konnectivity-agent/0.log"
Oct 30 13:11:25.275879 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:25.275853    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-228.ec2.internal_72d3c22f1a6da7d8f3667e690d2cfffa/haproxy/0.log"
Oct 30 13:11:32.105125 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:32.105035    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65c6d4fc99-x6wdt_4412a6ae-aaea-4aba-97f8-17ee44ff4dde/authorino/0.log"
Oct 30 13:11:32.714974 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:32.714941    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-7bbc6bb9d7-tvp9w_6c9068e7-5aea-467f-ad19-b6421524c6de/kuadrant-console-plugin/0.log"
Oct 30 13:11:33.783366 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:33.783333    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-68669df9c4-2nng6_17d543a6-06a9-46ac-9b53-e43b0c0b34f5/manager/0.log"
Oct 30 13:11:35.284971 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.284939    5837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"]
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285317    5837 cpu_manager.go:414] "containerMap: removing container" podUID="3929deb3-fa21-4e2f-8ada-6348c9faac3d" containerName="payload-processing"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285327    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3929deb3-fa21-4e2f-8ada-6348c9faac3d" containerName="payload-processing"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285340    5837 cpu_manager.go:414] "containerMap: removing container" podUID="f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" containerName="istio-proxy"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285346    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" containerName="istio-proxy"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285355    5837 cpu_manager.go:414] "containerMap: removing container" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285359    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285368    5837 cpu_manager.go:414] "containerMap: removing container" podUID="f855b07d-e71c-491e-b430-0d33bdd77a5d" containerName="istio-proxy"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285373    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f855b07d-e71c-491e-b430-0d33bdd77a5d" containerName="istio-proxy"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285381    5837 cpu_manager.go:414] "containerMap: removing container" podUID="4348ac37-952a-4000-ba42-3a8fa6f12c4d" containerName="revoke-keys"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285385    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4348ac37-952a-4000-ba42-3a8fa6f12c4d" containerName="revoke-keys"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285407    5837 cpu_manager.go:414] "containerMap: removing container" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerName="istio-proxy"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285412    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerName="istio-proxy"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285420    5837 cpu_manager.go:414] "containerMap: removing container" podUID="faf87439-5044-43cd-9b1a-200188299f24" containerName="istio-proxy"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285425    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf87439-5044-43cd-9b1a-200188299f24" containerName="istio-proxy"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285435    5837 cpu_manager.go:414] "containerMap: removing container" podUID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" containerName="istio-proxy"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285440    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" containerName="istio-proxy"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285445    5837 cpu_manager.go:414] "containerMap: removing container" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285450    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285455    5837 cpu_manager.go:414] "containerMap: removing container" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285460    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285466    5837 cpu_manager.go:414] "containerMap: removing container" podUID="ee36af90-164d-404c-be97-5dbaeafc0648" containerName="payload-pre-processing"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285471    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee36af90-164d-404c-be97-5dbaeafc0648" containerName="payload-pre-processing"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285476    5837 cpu_manager.go:414] "containerMap: removing container" podUID="bc1998d0-b85e-46aa-a6d4-92b828b66d08" containerName="payload-processing"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285481    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1998d0-b85e-46aa-a6d4-92b828b66d08" containerName="payload-processing"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285491    5837 cpu_manager.go:414] "containerMap: removing container" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285495    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy"
Oct 30 13:11:35.285494 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285507    5837 cpu_manager.go:414] "containerMap: removing container" podUID="25644850-37a6-4820-8ed2-4f00fedd8274" containerName="istio-proxy"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285511    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25644850-37a6-4820-8ed2-4f00fedd8274" containerName="istio-proxy"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285527    5837 cpu_manager.go:414] "containerMap: removing container" podUID="fb963087-5281-48f9-b322-110ae1590936" containerName="payload-pre-processing"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285531    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb963087-5281-48f9-b322-110ae1590936" containerName="payload-pre-processing"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285536    5837 cpu_manager.go:414] "containerMap: removing container" podUID="7b0047c7-c493-412a-83e1-10b48bdc06b6" containerName="revoke-keys"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285541    5837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0047c7-c493-412a-83e1-10b48bdc06b6" containerName="revoke-keys"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285600    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf87439-5044-43cd-9b1a-200188299f24" containerName="istio-proxy"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285608    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ddf5f4-5772-45d0-8fbb-f1eba1931608" containerName="istio-proxy"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285614    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="351759f4-8556-4412-952a-a9a63f93c23a" containerName="istio-proxy"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285619    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="089cb853-6545-4a2e-9aed-3b6cede912d6" containerName="main"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285625    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee36af90-164d-404c-be97-5dbaeafc0648" containerName="payload-pre-processing"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285630    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1998d0-b85e-46aa-a6d4-92b828b66d08" containerName="payload-processing"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285635    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ea72c6-dacd-42c2-a7c1-c32ea4135e66" containerName="istio-proxy"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285642    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="25644850-37a6-4820-8ed2-4f00fedd8274" containerName="istio-proxy"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285647    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="25644850-37a6-4820-8ed2-4f00fedd8274" containerName="istio-proxy"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285654    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb963087-5281-48f9-b322-110ae1590936" containerName="payload-pre-processing"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285659    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0047c7-c493-412a-83e1-10b48bdc06b6" containerName="revoke-keys"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285665    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="3929deb3-fa21-4e2f-8ada-6348c9faac3d" containerName="payload-processing"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285671    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9eaac99-ea1c-4fcb-b8e2-dc828e1089dd" containerName="istio-proxy"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285677    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6eaa6ea-8274-49fa-b007-753de046e268" containerName="main"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285682    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f855b07d-e71c-491e-b430-0d33bdd77a5d" containerName="istio-proxy"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285687    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4348ac37-952a-4000-ba42-3a8fa6f12c4d" containerName="revoke-keys"
Oct 30 13:11:35.286346 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.285692    5837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f24ec435-0ad6-4388-8a13-f38ca9a26800" containerName="istio-proxy"
Oct 30 13:11:35.289509 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.289488    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.306679 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.306648    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5tjtq\"/\"kube-root-ca.crt\""
Oct 30 13:11:35.306679 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.306648    5837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5tjtq\"/\"openshift-service-ca.crt\""
Oct 30 13:11:35.320522 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.320498    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"]
Oct 30 13:11:35.331540 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.331519    5837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5tjtq\"/\"default-dockercfg-rwqwj\""
Oct 30 13:11:35.396781 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.396757    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2-sys\") pod \"perf-node-gather-daemonset-qrwn6\" (UID: \"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.396912 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.396814    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2-podres\") pod \"perf-node-gather-daemonset-qrwn6\" (UID: \"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.396912 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.396896    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvf4g\" (UniqueName: \"kubernetes.io/projected/5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2-kube-api-access-qvf4g\") pod \"perf-node-gather-daemonset-qrwn6\" (UID: \"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.396991 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.396932    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2-proc\") pod \"perf-node-gather-daemonset-qrwn6\" (UID: \"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.397046 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.397033    5837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2-lib-modules\") pod \"perf-node-gather-daemonset-qrwn6\" (UID: \"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.498294 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.498267    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2-proc\") pod \"perf-node-gather-daemonset-qrwn6\" (UID: \"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.498465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.498335    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2-lib-modules\") pod \"perf-node-gather-daemonset-qrwn6\" (UID: \"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.498465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.498356    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2-sys\") pod \"perf-node-gather-daemonset-qrwn6\" (UID: \"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.498465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.498380    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2-proc\") pod \"perf-node-gather-daemonset-qrwn6\" (UID: \"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.498465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.498433    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2-podres\") pod \"perf-node-gather-daemonset-qrwn6\" (UID: \"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.498465 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.498459    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2-sys\") pod \"perf-node-gather-daemonset-qrwn6\" (UID: \"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.498657 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.498467    5837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvf4g\" (UniqueName: \"kubernetes.io/projected/5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2-kube-api-access-qvf4g\") pod \"perf-node-gather-daemonset-qrwn6\" (UID: \"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.498657 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.498474    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2-lib-modules\") pod \"perf-node-gather-daemonset-qrwn6\" (UID: \"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.498657 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.498502    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2-podres\") pod \"perf-node-gather-daemonset-qrwn6\" (UID: \"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.518675 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.518647    5837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvf4g\" (UniqueName: \"kubernetes.io/projected/5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2-kube-api-access-qvf4g\") pod \"perf-node-gather-daemonset-qrwn6\" (UID: \"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.561641 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.561588    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed/alertmanager/0.log"
Oct 30 13:11:35.601998 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.601760    5837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:35.683179 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.682959    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed/config-reloader/0.log"
Oct 30 13:11:35.802037 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.802013    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed/kube-rbac-proxy-web/0.log"
Oct 30 13:11:35.894857 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.894705    5837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"]
Oct 30 13:11:35.897180 ip-10-0-137-228 kubenswrapper[5837]: W1030 13:11:35.897151    5837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5799fbe5_7a5d_4a3f_ba95_393c6d0f3ad2.slice/crio-e82753dcbc17b4e2c2788064cfb94d595d950dc9a57358d4eacd33762da7477c WatchSource:0}: Error finding container e82753dcbc17b4e2c2788064cfb94d595d950dc9a57358d4eacd33762da7477c: Status 404 returned error can't find the container with id e82753dcbc17b4e2c2788064cfb94d595d950dc9a57358d4eacd33762da7477c
Oct 30 13:11:35.924367 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:35.924343    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed/kube-rbac-proxy/0.log"
Oct 30 13:11:36.048876 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:36.048842    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed/kube-rbac-proxy-metric/0.log"
Oct 30 13:11:36.178281 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:36.178217    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed/prom-label-proxy/0.log"
Oct 30 13:11:36.306836 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:36.306781    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cfac81cf-d52e-42f7-8bfd-5df2d9fdbaed/init-config-reloader/0.log"
Oct 30 13:11:36.583994 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:36.583961    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6" event={"ID":"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2","Type":"ContainerStarted","Data":"0ebea236566a875d9c4b9ab74e7b536686789584ab58e2ba564cdc9780a2aa92"}
Oct 30 13:11:36.584193 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:36.584021    5837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6" event={"ID":"5799fbe5-7a5d-4a3f-ba95-393c6d0f3ad2","Type":"ContainerStarted","Data":"e82753dcbc17b4e2c2788064cfb94d595d950dc9a57358d4eacd33762da7477c"}
Oct 30 13:11:36.584193 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:36.584061    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:36.590521 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:36.590488    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-gb5g9_f7ad82f0-86a1-43e3-8237-846a3aef0c88/kube-state-metrics/0.log"
Oct 30 13:11:36.624321 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:36.624281    5837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6" podStartSLOduration=1.6242681110000001 podStartE2EDuration="1.624268111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:36.623255618 +0000 UTC m=+2085.147539838" watchObservedRunningTime="2026-10-30 13:11:36.624268111 +0000 UTC m=+2085.148552329"
Oct 30 13:11:36.709942 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:36.709915    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-gb5g9_f7ad82f0-86a1-43e3-8237-846a3aef0c88/kube-rbac-proxy-main/0.log"
Oct 30 13:11:36.836576 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:36.836508    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-gb5g9_f7ad82f0-86a1-43e3-8237-846a3aef0c88/kube-rbac-proxy-self/0.log"
Oct 30 13:11:36.963828 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:36.963799    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6f9858f786-6dftr_7709b8a5-ca78-45d6-9c8c-136a13f50c72/metrics-server/0.log"
Oct 30 13:11:37.090565 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:37.090490    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-wqwgs_30a15bca-3504-49fe-9ecf-c9860108905e/monitoring-plugin/0.log"
Oct 30 13:11:37.221953 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:37.221928    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9gvsj_70b50531-ed83-4bac-bdf1-0992042e01a0/node-exporter/0.log"
Oct 30 13:11:37.345736 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:37.345664    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9gvsj_70b50531-ed83-4bac-bdf1-0992042e01a0/kube-rbac-proxy/0.log"
Oct 30 13:11:37.467319 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:37.467294    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9gvsj_70b50531-ed83-4bac-bdf1-0992042e01a0/init-textfile/0.log"
Oct 30 13:11:38.489500 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:38.489473    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-ql52w_a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef/kube-rbac-proxy-main/0.log"
Oct 30 13:11:38.618804 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:38.618777    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-ql52w_a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef/kube-rbac-proxy-self/0.log"
Oct 30 13:11:38.743738 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:38.743668    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-ql52w_a7eb0dd3-f31d-4165-a7f8-8ec9594d1cef/openshift-state-metrics/0.log"
Oct 30 13:11:38.881737 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:38.881711    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a73f77c-5563-4a51-94ab-2febe951313c/prometheus/0.log"
Oct 30 13:11:39.020935 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:39.020858    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a73f77c-5563-4a51-94ab-2febe951313c/config-reloader/0.log"
Oct 30 13:11:39.028846 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:39.028820    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"]
Oct 30 13:11:39.040432 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:39.040389    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e32c90f-hzj97"]
Oct 30 13:11:39.042168 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:39.042147    5837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"]
Oct 30 13:11:39.043675 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:39.043653    5837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54e7adec-jwpqb"]
Oct 30 13:11:39.148570 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:39.148536    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a73f77c-5563-4a51-94ab-2febe951313c/thanos-sidecar/0.log"
Oct 30 13:11:39.272745 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:39.272672    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a73f77c-5563-4a51-94ab-2febe951313c/kube-rbac-proxy-web/0.log"
Oct 30 13:11:39.403222 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:39.403201    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a73f77c-5563-4a51-94ab-2febe951313c/kube-rbac-proxy/0.log"
Oct 30 13:11:39.529184 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:39.529114    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a73f77c-5563-4a51-94ab-2febe951313c/kube-rbac-proxy-thanos/0.log"
Oct 30 13:11:39.603953 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:39.603928    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf688ee-b349-40af-a2cc-14877d335e34" path="/var/lib/kubelet/pods/1cf688ee-b349-40af-a2cc-14877d335e34/volumes"
Oct 30 13:11:39.604337 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:39.604323    5837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6879e6-651c-4a62-b0fa-511232557e12" path="/var/lib/kubelet/pods/5c6879e6-651c-4a62-b0fa-511232557e12/volumes"
Oct 30 13:11:39.654108 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:39.654088    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a73f77c-5563-4a51-94ab-2febe951313c/init-config-reloader/0.log"
Oct 30 13:11:40.151333 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:40.151308    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7496d6d446-bnssg_b6118a33-9c2f-42e9-bd9a-27e350e26d7b/telemeter-client/0.log"
Oct 30 13:11:40.272199 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:40.272147    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7496d6d446-bnssg_b6118a33-9c2f-42e9-bd9a-27e350e26d7b/reload/0.log"
Oct 30 13:11:40.391202 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:40.391168    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7496d6d446-bnssg_b6118a33-9c2f-42e9-bd9a-27e350e26d7b/kube-rbac-proxy/0.log"
Oct 30 13:11:40.518091 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:40.518066    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56878775cd-4pz7n_9ecfafe4-94c0-451e-b646-54aa9dce7f46/thanos-query/0.log"
Oct 30 13:11:40.638738 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:40.638711    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56878775cd-4pz7n_9ecfafe4-94c0-451e-b646-54aa9dce7f46/kube-rbac-proxy-web/0.log"
Oct 30 13:11:40.775468 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:40.775383    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56878775cd-4pz7n_9ecfafe4-94c0-451e-b646-54aa9dce7f46/kube-rbac-proxy/0.log"
Oct 30 13:11:40.898147 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:40.898121    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56878775cd-4pz7n_9ecfafe4-94c0-451e-b646-54aa9dce7f46/prom-label-proxy/0.log"
Oct 30 13:11:41.016504 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:41.016468    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56878775cd-4pz7n_9ecfafe4-94c0-451e-b646-54aa9dce7f46/kube-rbac-proxy-rules/0.log"
Oct 30 13:11:41.136038 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:41.135975    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56878775cd-4pz7n_9ecfafe4-94c0-451e-b646-54aa9dce7f46/kube-rbac-proxy-metrics/0.log"
Oct 30 13:11:42.601503 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:42.601470    5837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-qrwn6"
Oct 30 13:11:45.093174 ip-10-0-137-228 kubenswrapper[5837]: E1030 13:11:45.093135    5837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dg6tk" containerName="extractor"
Oct 30 13:11:50.157064 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:50.157036    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zh8sx_2343d79d-4146-4f1f-90be-ad84a256a4a5/dns/0.log"
Oct 30 13:11:50.294766 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:50.294739    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zh8sx_2343d79d-4146-4f1f-90be-ad84a256a4a5/kube-rbac-proxy/0.log"
Oct 30 13:11:50.561205 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:50.561175    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-84tjf_e4fe0229-c475-400a-b6c0-48d80476b3b5/dns-node-resolver/0.log"
Oct 30 13:11:51.990450 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:51.990416    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 13:11:51.995181 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:51.995161    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9trzv_0a195064-4c82-47ec-939e-8f0ac45fb0f6/ovn-acl-logging/0.log"
Oct 30 13:11:52.406451 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:52.406355    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p9gs7_c72d4793-688e-4c37-bb51-e0a9100a2203/node-ca/0.log"
Oct 30 13:11:54.218713 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:54.218675    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kvdnb_c5613511-ebe4-44bb-9d1b-a0652e3f97b8/discovery/0.log"
Oct 30 13:11:54.325921 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:54.325889    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-dc45784b6-t2t85_5d0188ec-41eb-45b4-ae5c-705341675edf/kube-auth-proxy/0.log"
Oct 30 13:11:54.524209 ip-10-0-137-228 kubenswrapper[5837]: I1030 13:11:54.524184    5837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-xhs8n_99edc7cd-e5d4-4ac9-ab07-c0b91926a654/istio-proxy/0.log"